Month: July 2018

DBMS, Scripting, Git experiences on Mac – (Very belated) Interlude

Over the  (2016) Christmas holidays, I started to try to work on the UQ course chooser. But for some reason, I never finished posting the blog. Sigh… So, here are a few random thoughts and some of the issues I encountered while working on the project at the very beginning, with only my MacBook Pro to develop on.

Git on Mac

  • Git on Windows is actually easier because the git bash app doesn’t remember the password. Git on Mac tries to use Apple Keychain to remember the password but it wouldn’t let me actually enter the password. Even in Unix with the hidden password entry. I ended up installing SourceTree to manage the git repositories on my Mac laptop.
    • On Windows, I’m still using Git Bash which now has broken so I have had to resort to using Git inside of the Linux Subsystem for Windows.
    • Though of course, I can still use Git inside PyCharm, my IDE of choice so far.

Now, about a year later I’ve been doing a lot more development on Mac and seem to have fixed the login issues. (I think I got a new version from Git and that was better.) I’m still using SourceTree because it’s easier to see the differences.)

MySQL on Mac

  • The Windows installer included everything. The Mac installer separates out the Workbench from the Server and Utilities. I first installed the Workbench cause I that’s all I needed initially, then I wanted to use the database and of course the database server itself wasn’t installed so I had to instal that separately.
  • After that I still had problems getting it working on the command line. The installer didn’t add it to $PATH – I had to look up what was wrong, and how to fix it manually. Following instructions on the internet, I had to add the symlink /usr/local/mysql/bin to $PATH
  • http://stackoverflow.com/questions/7703041/editing-path-variable-on-mac
  • Then have to reset the default password
  • ALTER USER ‘root’@’localhost’ IDENTIFIED BY [password];
  • https://dev.mysql.com/doc/refman/5.7/en/alter-user.html

Now, just now, when I tried to run queries in MySQL Workbench, the application wouldn’t even run a simple query. When I used Google to search for [mysql workbench crashes on query], the first result directed me to install the later version, then I just used the built-in Check For Updates and got the most recent update. This fixed my problem too, now it works fine again.

Python Database stuff

Have to get the MySQL drivers for Python from here. https://dev.mysql.com/downloads/connector/python/2.1.html

http://stackoverflow.com/questions/23376103/python-3-4-0-with-mysql-database

After much wailing and gnashing of teeth (well, a bit anyway) I just decided to use PyMySQL with is a pure Python MySQL module – apparently, it’s slower than using a C library, but it doesn’t have dependencies and just works. I had many issues getting MySQLdb or the fork, MySQL client to work on either Windows or Mac. For the size of my database, I’m not too worried about the speed of the implementation, particularly given that the main bottleneck will be the web scraping anyway.
Originally used the built-in Python HTML scraping library, but quickly changed to using Beautifulsoup. It is much more flexible and easier to write.

Of course, I actually got much further in the project as you can read about in the earlier posts, and also I probably shouldn’t let blogs like this go too long I suppose. It’s much easier to blog about the successes rather than the issues. If you have any ideas to make this easier… please let me know 🙂

Posted by Anthony in Course Chooser

Finished Another Course!

The end of this semester I finished COMS7201 – Computer Networks 1.  For some reason, it was one of the most satisfying courses I’ve taken.

I’ve written about all about it right here – Computer Networks 1, describing the assessment and several of the main concepts taught. Since writing that page, I still haven’t gone back and fixed the issues in the assignments, particularly Assignment 3. I want to implement the 1s Complement checksum and totally native IP header generation.  I’ve been doing other things with my spare time instead, like adding a feature to my Chat Log Parser/Viewer to group chats by day, writing for Weekend Notes on Mt Gambier plus other life skills updating and of course general life.

One approach I’ve started a few weeks ago is to create a daily goal to practice coding (and I’ve added writing) every day, but only for 15 minutes each day. This forces me to break up the task into discrete units of work – one (or two) unit test/s, one (or 2) method/s or classes, one paragraph or just the outline. I’ve found this method to work well so far, I don’t get put off with a long session when I want to relax but I still make progress. If I find I’m going really well and get into the groove (Flow), I can certainly keep going, but if not I can stop.

The above approach doesn’t work so well with university study because the assignments have due dates. This is good for personal projects and hobbies though.

Posted by Anthony

Day Grouping in the Chat Log Parser!

Finally, I’ve added a feature to my chat log parser to group messages by day. It’s something I’ve wanted to be able to do for a while now. This took a bit of work to add a new data structure and update the HTML Jinja2 templates. But now the base of it is done and if I want, I can customise the styles.

Screenshot of Chat Log with day grouping

Oh hey Sunny!

Originally, I tried to do the day grouping inside the Jinja2 templates, but this proved rather difficult – I ended up spending hours in the library trying to work out nested loops inside a templating language. Eventually, I realised I should perform the day grouping on the data structure instead, so I just loop over each day, then created the minute bundles inside.

Technical Info!

Creating the day grouping in the ChatLog data structure turned out to be surprisingly easy:

Continue reading →

Posted by Anthony in chat log parser