Peter Jalbert (Work Log)

09/27/2016 15:00-18:00: Set up Staff wiki page, work log page; registered for Slack, Microsoft Remote Desktop; downloaded Selenium on personal computer, read Selenium docs. Created wiki page for Moroccan Web Driver Project.

09/29/2016 15:00-18:00: Re-enroll in Microsoft Remote Desktop with proper authentication, set up Selenium environment and Komodo IDE on Remote Desktop, wrote program using Selenium that goes to a link and opens up the print dialog box. Developed computational recipe for a different approach to the problem.

09/30/2016 12:00-14:00: Selenium program selects view pdf option from the website, and goes to the pdf webpage. Program then switches handle to the new page. CTRL S is sent to the page to launch save dialog window. Text cannot be sent to this window. Brainstorm ways around this issue. Explored Chrome Options for saving automatically without a dialog window. Looking into other libraries besides selenium that may help.

10/3/2016 13:00 - 16:00: Moroccan Web Driver projects completed for driving of the Monarchy proposed bills, the House of Representatives proposed bills, and the Ratified bills sites. Begun process of devising a naming system for the files that does not require scraping. Tinkered with naming through regular expression parsing of the URL. Structure for the oral questions and written questions drivers is set up, but need fixes due to the differences in the sites. Fixed bug on McNair wiki for women's biz team where email was plain text instead of an email link. Took a glimpse at Kuwait Parliament website, and it appears to be very different from the Moroccan setup.

10/6/2016 13:30 - 18:00: Discussed with Dr. Elbadawy about the desired file names for Moroccan data download. The consensus was that the bill programs are ready to launch once the files can be named properly, and the questions data must be retrieved using a web crawler which I need to learn how to implement. The naming of files is currently drawing errors in going from arabic, to url, to download, to filename. Debugging in process. Also built a demo selenium program for Dr. Egan that drives the McNair blog site on an infinite loop.

10/7/2016 12:00 - 14:00: Learned unicode and utf8 encoding and decoding in arabic. Still working on transforming an ascii url into printable unicode.

10/11/2016 15:00 - 18:00: Fixed arabic bug, files can now be saved with arabic titles. Monarchy bills downloaded and ready for shipment. House of Representatives Bill mostly downloaded, ratified bills prepared for download. Started learning scrapy library in python for web scraping. Discussed idea of screenshot-ing questions instead of scraping.

10/13/2016 13:00-18:00: Completed download of Moroccan Bills. Working on either a web driver screenshot approach or a webcrawler approach to download the Moroccan oral and written questions data. Began building Web Crawler for Oral and Written Questions site. Edited Moroccan Web Driver/Crawler wiki page. Moroccan Web Driver

10/14/2016 12:00-14:00: Finished Oral Questions crawler. Finished Written Questions crawler. Waiting for further details on whether that data needs to be tweaked in any way. Updated the Moroccan Web Driver/Web Crawler wiki page. Moroccan Web Driver

10/18/2016 15:00-18:30: Finished code for Oral Questions web driver and Written Questions web driver using selenium. Now, the data for the dates of questions can be found using the crawler, and the pdfs of the questions will be downloaded using selenium. Moroccan Web Driver

10/20/2016 13:00-18:00: Continued to download data for the Moroccan Parliament Written and Oral Questions. Updated Wiki page. Started working on Twitter project with Christy. Moroccan Web Driver

10/21/2016 12:00-14:00: Continued to download data for the Moroccan Parliament Written and Oral Questions. Looked over Christy's Twitter Crawler to see how I can be helpful. Dr. Egan asked me to think about how to potentially make multiple tools to get cohorts and other sorts of data from accelerator sites. See Accelerator List He also asked me to look at the GovTrack Web Crawler for potential ideas on how to bring this project to fruition.

11/1/2016: 15:00-18:00: Continued to download Moroccan data in the background. Went over code for GovTracker Web Crawler, continued learning Perl. GovTrack Web Crawler Began Kuwait Web Crawler/Driver.

11/3/2016: 13:00-18:00: Continued to download Moroccan data in the background. Dr. Egan fixed systems requirements to run the GovTrack Web Crawler. Made significant progress on the Kuwait Web Crawler/Driver for the Middle East Studies Department.

11/4/2016: 12:00-14:00: Continued to download Moroccan data in the background. Finished writing initial Kuwait Web Crawler/Driver for the Middle East Studies Department. Middle East Studies Department asked for additional embedded files in the Kuwait website. Moroccan Web Driver

11/8/2016: 15:00-18:00: Continued to download Moroccan data in the background. Finished writing code for the embedded files on the Kuwait Site. Spent time debugging the frame errors due to the dynamically generated content. Never found an answer to the bug, and instead found a workaround that sacrificed run time for the ability to work. Moroccan Web Driver

11/10/2016 13:00-18:00: Continued to download Moroccan data and Kuwait data in the background. Began work on Google Scholar Crawler. Wrote a crawler for the Accelerator Project to get the HTML files of hundreds of accelerators. The crawler ended up failing; it appears to have been due to HTTPS.

11/11/2016 12:00-2:00: Continued to download Moroccan data in the background. Attempted to find bug fixes for the Accelerator Project crawler.

11/15/2016 15:00-18:00: Finished download of Moroccan Written Question pdfs. Wrote a parser with Christy to be used for parsing bills from Congress and eventually executive orders. Found bug in the system Python that was worked out and rebooted.

11/17/2016 13:00-18:00: Wrote a crawler to retrieve information about executive orders, and their corresponding pdfs. They can be found here. Next step is to run code to convert the pdfs to text files, then use the parser fixed by Christy.