American Community Survey (ACS) Data - Revision history

Ed: /* Working with the API */

2021-02-15T17:54:21Z

Working with the API

Ed: /* Working with the API */

2021-02-15T17:53:55Z

Working with the API

Ed: /* Notes */

2021-02-14T00:31:24Z

Notes

Ed: /* Working with the API */

2021-02-13T02:12:44Z

Working with the API

Ed at 02:11, 13 February 2021

2021-02-13T02:11:57Z

Ed: /* get_ACS.py */

2021-02-13T00:01:42Z

get_ACS.py

Ed at 23:26, 12 February 2021

2021-02-12T23:26:46Z

Ed: /* get_ACS.py */

2021-02-12T20:21:54Z

get_ACS.py

Ed: /* Working with the API */

2021-02-12T20:19:20Z

Working with the API

Ed: /* Using api.census.gov */

2021-02-12T18:33:20Z

Using api.census.gov

← Older revision		Revision as of 17:54, 15 February 2021
Line 69:		Line 69:
	==Working with the API==		==Working with the API==

		+	===Getting a Key===
	The API doesn't seem to require a key, but you can request one from https://api.census.gov/.		The API doesn't seem to require a key, but you can request one from https://api.census.gov/.

@@ Line 71: / Line 71: @@
 The API doesn't seem to require a key, but you can request one from https://api.census.gov/.
-The data contains null values, as well as values that indicate error codes (-999999999,-888888888,-666666666,-555555555,-222222222, and others. See https://www.census.gov/data/developers/data-sets/acs-1year/notes-on-acs-estimate-and-annotation-values.html. Generally, it appears that all negative values can be striped. Every field is string quoted (even if it's a number) including error codes, except for nulls.
+===Quoting, Nulls, and Error Codes===
 The API uses improper URL encoding (or more accurately, doesn't use URL encoding), so most methods for auto-populating URL parameters in scripting languages will not function correctly. Build URLs manually.
 The API returns improper JSON. Essentially, it just encodes a flat data structure into JSON-like syntax one line at a time, and completely ignores the structure of the data (and so much of the point of encoding it in JSON!). However, the first (zeroth) column is the requested geography name, so multiple pulls can be joined using that. And the first (zeroth) row is a valid set of column headers. Also, the nulls are valid JSON encodings. This, and the strong quoting, makes using a JSON handler likely worthwhile. An alternative is to strip the JSON tags and treat the data as a badly encoded CSV, which it is.
-To request a group ask for '''group(B06009)''' where you would normally request a field like '''B06011_001E'''. It looks like programmers can rely on groups and fields being identifiable from the "_" and the string length (\w{1}\d{5}\w{0,1}_\d{3}\w{1}).
+===Field and Group Names===
 =Old Approach using AFF=

@@ Line 13: / Line 13: @@
 =Notes=
-The ACS releases datasets in October of the following year. It has the following datasets:
+The [https://www.census.gov/programs-surveys/acs/news/data-releases/2019/release-schedule.html ACS releases datasets] in September through December of the following year. It has the following datasets:
 *ACS5: 5-year estimates for a broad list of variables, from 2009-2020, covering 29,573 places.
 *ACS1: 1-year estimates from 2005-2020 for select variables for 630 places with populations of 65,000 people or more
 *ACS1s: 1-year supplements covering 2,323 places with 20,000 people or more
 ACS provides information [https://www.census.gov/programs-surveys/acs/geography-acs/areas-published.html on a broad range of geographies, including states, places, and zctas].
@@ Line 24: / Line 25: @@
 *PUMS data for 2017: https://www2.census.gov/programs-surveys/acs/data/pums/2017/1-Year/
 *Population and other data are available from: http://census.ire.org/data/bulkdata.html
 Note: PUMS is only available for places in the [https://en.wikipedia.org/wiki/Public_Use_Microdata_Area Public Use Microdata Areas], which have a population 100,000 or more.
 =Using api.census.gov=

@@ Line 70: / Line 70: @@
 The API doesn't seem to require a key, but you can request one from https://api.census.gov/.
-The data contains null values, as well as values that indicate error codes (-999999999,-888888888,-666666666,-555555555,-222222222, and others. See https://www.census.gov/data/developers/data-sets/acs-1year/notes-on-acs-estimate-and-annotation-values.html. Generally, it appears that all negative values can be striped. Every field is string quoted including error codes, except for nulls.
+The data contains null values, as well as values that indicate error codes (-999999999,-888888888,-666666666,-555555555,-222222222, and others. See https://www.census.gov/data/developers/data-sets/acs-1year/notes-on-acs-estimate-and-annotation-values.html. Generally, it appears that all negative values can be striped. Every field is string quoted (even if it's a number) including error codes, except for nulls.
 The API uses improper URL encoding (or more accurately, doesn't use URL encoding), so most methods for auto-populating URL parameters in scripting languages will not function correctly. Build URLs manually.

@@ Line 9: / Line 9: @@
 *[[PostGIS Installation]]
 *[[Urban Start-up Agglomeration]]
 =Notes=
@@ Line 23: / Line 24: @@
 *PUMS data for 2017: https://www2.census.gov/programs-surveys/acs/data/pums/2017/1-Year/
 *Population and other data are available from: http://census.ire.org/data/bulkdata.html
 Note: PUMS is only available for places in the [https://en.wikipedia.org/wiki/Public_Use_Microdata_Area Public Use Microdata Areas], which have a population 100,000 or more.

@@ Line 45: / Line 45: @@
   E:\projects\census\
-An example command is:
+Example commands are:
-  python3 get_ACS.py B06011_001E B06009 B02001
+  python3 get_ACS.py B19013_001E B06009 B02001
 The script produces tab-delimited text files (to change to csv, edit line 63) with timestamped filenames.

@@ Line 12: / Line 12: @@
 =Notes=
-The ACS releases datasets in October of the following year. It has 1-year estimates or 5 year estimates. The five year estimates have more information.
+The ACS releases datasets in October of the following year. It has the following datasets:
-The 1 yr estimates are only available for places with populations of 65,000 people or more, the 1yr supplemental estimates cover down to places with 20,000 people or more, and the 5 yr estimates cover a whopping 29,573 places [https://www.census.gov/programs-surveys/acs/geography-acs/areas-published.html], as compared with just 630 for the 1yr and 2,323 or the 1yr supplements.
+ACS provides information [https://www.census.gov/programs-surveys/acs/geography-acs/areas-published.html on a broad range of geographies, including states, places, and zctas].
-One can download the PUMS data for 2017 here: https://www2.census.gov/programs-surveys/acs/data/pums/2017/1-Year/ However, PUMS is only available for places in the [https://en.wikipedia.org/wiki/Public_Use_Microdata_Area Public Use Microdata Areas], which have a population 100,000 or more.
+==Other data sources==
-And state-by-state, it looks possible to get the latest data profiles (2013-2017 5 yr estimates) but not place by place as data, just as an aggregate:
+*PUMS data for 2017: https://www2.census.gov/programs-surveys/acs/data/pums/2017/1-Year/
-https://www.census.gov/acs/www/data/data-tables-and-tools/data-profiles/
+*Population and other data are available from: http://census.ire.org/data/bulkdata.html
-For just population, also see: http://census.ire.org/data/bulkdata.html. This seems to be based on the 2000 or 2010 census data only.
+=Using api.census.gov=
-=Using api.census.gov=
+The new approach is to use api.census.gov to request information. This applies to ACS and a broad range of other Census datasets.
-The new approach is to use api.census.gov to request information. Each year has an API URL, which specifies the dataset and that takes '''get''' and '''for''' name-value pairs that specify variables and geographies, respectively. Generally, one can request geography, variable, and examples HTML pages on the same URL (rather than specifying the params). The ACS5 data goes back to 2009 and the ACS1 goes back to 2005 using the base URL: '''https://api.census.gov/data/yyyy/acs/acsx/''', where yyyy is year and x is 1 or 5.
+Pages listing available geographies, variables, and some examples are on the same base URL. For, example:
 ==get_ACS.py==
@@ Line 39: / Line 50: @@
 The script produces tab-delimited text files (to change to csv, edit line 63) with timestamped filenames.
-Note: In the script, I rely on the response.json() method of request to handle the "JSON" produced by the API. I never use json.loads or similar. The Census doesn't adhere to any reasonable standards for encoding data.
+Note: In the script, I rely on the response.json() method of request to handle the "JSON" produced by the API. I never use json.loads or similar. The Census doesn't adhere to any reasonable standards for encoding URLs or data!
 ==Examples and Documentation==
 Example API calls for median income, educational attainment, and race from the 5yr ACS, using the 2019 dataset:

← Older revision		Revision as of 18:33, 12 February 2021
Line 26:		Line 26:

	The new approach is to use api.census.gov to request information. Each year has an API URL, which specifies the dataset and that takes '''get''' and '''for''' name-value pairs that specify variables and geographies, respectively. Generally, one can request geography, variable, and examples HTML pages on the same URL (rather than specifying the params). The ACS5 data goes back to 2009 and the ACS1 goes back to 2005 using the base URL: '''https://api.census.gov/data/yyyy/acs/acsx/''', where yyyy is year and x is 1 or 5.		The new approach is to use api.census.gov to request information. Each year has an API URL, which specifies the dataset and that takes '''get''' and '''for''' name-value pairs that specify variables and geographies, respectively. Generally, one can request geography, variable, and examples HTML pages on the same URL (rather than specifying the params). The ACS5 data goes back to 2009 and the ACS1 goes back to 2005 using the base URL: '''https://api.census.gov/data/yyyy/acs/acsx/''', where yyyy is year and x is 1 or 5.
		+
		+	==get_ACS.py==
		+
		+	A script called '''get_ACS.py''' automates the retrieval and compiling of results from api.census.gov. It has the year, geographic unit, and dataset in a (user-editable) parameters dictionary and is called with variables as command line parameters. Note that groups of variables are called in the same way as individual variables. An example command is:
		+	python3 get_ACS.py B06011_001E B06009 B02001
		+
		+	==Examples and Documentation==

	For, example, for 2019, the URLs are:		For, example, for 2019, the URLs are: