The Right Tools for the Job

I’ve been involving myself in a few projects lately, for both work and personal life. During these projects, I’ve been thinking about what tools and technologies I use, and to use while I’m in the planning process. I’d like to share some of these.

Updated: 29th March 2018 – Git Bash for Mac

Continue reading →

Posted by Anthony

Auto-updating Shared Hosting from Git

I’ve found that when I update my website, it’s tedious to manually copy the files up to my CPanel hosting platform. And my website source code is already on Github version control, so I wanted to be able to automatically update the static files when I push my HTML up to Github.

The only way to achieve this is to utilise Git’s post-receive hook (and Github’s webhooks) and trigger some code on my site which could a perform git pull.

Git hooks are actions that can run your own scripts when events happen to your repository – commit or pull for example – then do whatever task is handy for you. Webhooks are HTTP payloads sent from one server to another, when some event occurs, to trigger off another action in turn on the receiving server.

Some research led me to oxguy3’s deploy.php Github Gist which receives Github’s post-receive webhook and initiates a git pull on on the web host:

Oxguy3’s deploy.php script.

Set up a webhook on your repository in the settings of the Github repository and give it a password. Really I just followed oxguy3’s directions, it’s well documented.

Their PHP script is quite good, it even uses Github’s authorisation headers to prevent randoms from triggering the deploy script.

But oxguy3’s script had to ssh to their server (inside the script) since their host had blocked PHP shell_exec. But my web host hasn’t blocked shell_exec so I could take out those pieces of code. So my forked gist becomes:

My deploy.php script

The script also emails me when it’s complete and the output. If you started from an empty public_html folder you’d be right to run the script now. (I wasn’t though, read on if interested.)

Continue reading →
Posted by Anthony, 0 comments

I’ve graduated!

Finishing semester 2 2018 and passing both COMS7003 – Information Security and COMP7506 – Algorithms and Data Structures this semester now gives me a Graduate Certificate in Information Technology

I have a piece of paper to prove my learning too!

So this whole degree I have done 9 subjects:

  1. CSSE7030 – Introduction to Software Engineering
  2. INFS7900 – Introduction to Information Systems
  3. INFS7903 – Relational Database Systems
  4. INFS7204 – Services Oriented Architecture
  5. CSSE7202 – Advanced Software Engineering
  6. INFS7202 – Web Information Systems
  7. COMS7201 – Computer Networks 1
  8. COMS7003 – Information Security
  9. COMP7505 – Algorithms and Data Structures

I’ve been wondering this semester which subjects have I found the most useful or enjoyable or interesting. It’s really hard to say if I’m going to be objective about it. The first 2 certainly gave me the biggest jump in knowledge – from almost nothing to actually being a programmer and building databases!

INFS7204, COMS7201, and CSSE7202 probably gave me the next biggest jump in knowledge and practical skills. While the SOA side of INFS7204 may not be so relevant these days, the overall idea of spreading functionality over simple services is very much relevant still, and the experience with C# was good. CSSE7202 taught some programming concepts which explained some of the workings we experienced in C#. Whilst I may not need the in-depth understanding of TCP, the overall networking experience and theory from COMS7201 was quite interesting and I think it’s quite useful in this day and age. That subject gave me experience in network programming in Python at the byte and binary level, as well as my first introduction to base 2 and hexadecimal notation.

Algorithms and Data Structures is a hard one to discuss – while I hated the course, I learnt a lot of things, got more experience in object oriented programming and recursion, and well graduated earlier by doing 2 subjects this semester.

Information Security was quite an interesting course and well taught, I think it covered a range of topics that are useful in the modern IT field. The assignments were satisfying at least.

Web Information Systems and Relational Database Systems were probably the least useful subjects. Unfortunately I probably could have learnt much the same content in WIS for free online – though it was a good chance to build a real web application from scratch. Relational Database Systems was interesting, but honestly I find it hard to relate the knowledge back to writing better SQL – if I was going to work for Oracle or work on PostgreSQL maybe it would be useful. But still learning is and these 2 were part of the overall degree anyway.

Trying to remain objective, I think my Grad Dip IT was a good choice and has been good for me overall. I think a shorter course like a Graduate Certificate or Graduate Diploma is a good way to break into IT. Maybe you could to a bootcamp and combine with some university study for theory and background too. I won’t get into that further here though.

Emotionally, I suppose these 3 years of part-time study has been good for my resilience with some rather stressful (for me at least) times, which I won’t get into. Suffice to say, what “they” say about pushing yourself is true, you do grow when you move out of your comfort zone.

Posted by Anthony, 0 comments

Conditional but static text in Razor Pages

On a recent work project I wanted to display some static text on an Razor Page, but only if a certain field in the database was empty or whitespace. So, if the entity’s title wasn’t set in the database, display a generic title in the web front-end.

I tried using a normal if … else block but had trouble with it because the text was supposed to be inside all inside of the HTML title element and Razor Pages was getting confused (as was I) when I tried to include static text inside the if block and  C# code inside the else block.

Continue reading →

Posted by Anthony in Learnings on Mac

At work I have an Core 2.1 web app I’m developing. At work we use Windows and Visual Studio 2017 to develop on with a local MS SQL Server as the database backend. But, I want to be able to work on the project away from the office and I only have a Mac laptop. So I’m trying to get it running on Mac, which should be possible since it’s .Net Core after all. There were a few things I needed to change, which turned out to be one line!

Continue reading →

Posted by Anthony

Django Project Critical Evaluation

In my earlier post, I was installing Django on Mac OS X by following their own tutorial partly to get a feel for the database site of the MVC framework. This was to evaluate whether I should use an off-the-shelf framework to just build my own. My requirements are:

  • Easy to code.
  • Preferably easy to manage the database and existing data.
  • Preferably users, security etc for free.
  • Reusable View components for menus etc.
  • Quality framework-created database table structure.

I don’t want to write forms to manage the data or all the user and authentication and session stuff. I can, but I don’t want to.

After I followed the tutorial and inspected the database generated I could critically evaluate the framework.

Django’s model-generated-data

  • Logically structured OK
  • Table names are the same as the entity in the model, but by default prepended with the Django app name and lower-cased. (You can change the table name in the model.)
  • Foreign-keys are generated and handled sensibly.
  • There is also a Shell – which gives you programmatic access to the data model. It’s basically just a standard Python REPL with some extra enhancements for Django stuff.

Django Shell

Django’s Admin Site

  • supports multiple users
  • Seems to keep a history of the changes (the meta-data not the actual changes) as made in the Admin site.
  • Intelligently builds the forms based on the data type in the model – dates have date pickers, strings have text inputs etc.

Django Admin

Views/URL Mappings

  • Can be as simple as a return statement or return a render() method for an HTML template or any other Python library as long as it can fit into the HTTP response – JSON/PDF/HTML/media/whatever… From the website:

“Your view can read records from a database, or not. It can use a template system such as Django’s – or a third-party Python template system – or not. It can generate a PDF file, output XML, create a ZIP file on the fly, anything you want, using whatever Python libraries you want.

All Django wants is that HttpResponse. Or an exception.”

  • URLs and parameters are mapped in a separate file which ‘captures’ parts of the GET request and converts parts of it to Python variables to pass along to the appropriate view method. It seems like you can have heaps of view methods all contained in different Python files, it’s just standard Python syntax to call things and pass data around. Possibly similar to other frameworks like which also has URL mappings.

Things I’m not sure about yet

  • Can you password protect an entire site? Like for an internal app?
  • I read somewhere about reverse engineering an existing database or even generating an ER diagram based on Django’s model. These sound pretty cool if possible.


  • The Django Project’s tutorial is one of the best getting-started tutorials I’ve used, props to them.  They must have tested it well, on various audiences. Only thing is some parts seem like suggestions, but it turns out later steps refer to code given so I had to go back. But if you did everything shown, you wouldn’t have this problem.
  • Programmatic testing is built right into Django, I’m surprised that you can even test Views. Not necessarily the HTML output, but at least whether or not a view gives you a particular HTTP response code – they give the example of permissions or hidding polls without questions.

I’ll finish the testing tutorial and static files tutorial later, but so far Django seems a pretty capable MVC framework and I’ll probably use it for at least some of the app ideas I have.

Posted by Anthony

Installing Django on Mac OS X

I’m writing this partially for myself so I have a reference for what’s wrong while trying to install Django in a Python virtual environment on OS X. I’m just going to write about errors I encountered I think.

Create the virtualenv

In the directory where I wanted to put the virtual environment type:

virtualenv django-ve

where django-ve will be the virtual environment. It should exist first.

Activate it:

cd django-ve
source bin/activate

Install Django

Following Django’s own guide still use pip to install it:

pip install Django

But when I did this I got an error with the SSL connection to PyPy exactly as in, this Stack Overflow post. The solution was as they describe in the accepted answer though you also have to update pip both inside the new virtual environment and outside. (I first tried updating it, then recreating the virtual environment, but this didn’t help, which kind of makes sense if it fetches a new pip each time.)

Install the database drivers

Since I want to use a full MySQL database I needed to install the Python connectors. I stuck with the Django recommended way. I tried to follow their instructions.

When I tried to install mysqlclient I got this same error. Updating Wheels as per one of the suggestions didn’t help, but running

xcode-select --install

did fix the problem by installing the Apple provided developer command line tools which install a C compiler.


I verified that the Python virtualenv could see Django as instructed and it could.

At this point, the setup guide ended so I’m just proceeding along with the tutorial.


I continued through the tutorial pretty well OK up until the part about configuring timezones. Silly me, I wrongly assumed I could modify it from ‘UTC’ to ‘AEST’, but even as is linked in their own guide, it’s based on the TZ Database so I should use ‘Australia/Queensland’

Then proceeding along and running

python migrate

runs successfully and going back to MySQL Workbench I can see all the tables that Django created. Looking good so far.

By this stage I’ve set it up and worked connect it to a real database, so that’s enough for now. I may come back later and critically evaluate Django for my purposes.

Posted by Anthony

DBMS, Scripting, Git experiences on Mac – (Very belated) Interlude

Over the  (2016) Christmas holidays, I started to try to work on the UQ course chooser. But for some reason, I never finished posting the blog. Sigh… So, here are a few random thoughts and some of the issues I encountered while working on the project at the very beginning, with only my MacBook Pro to develop on.

Git on Mac

  • Git on Windows is actually easier because the git bash app doesn’t remember the password. Git on Mac tries to use Apple Keychain to remember the password but it wouldn’t let me actually enter the password. Even in Unix with the hidden password entry. I ended up installing SourceTree to manage the git repositories on my Mac laptop.
    • On Windows, I’m still using Git Bash which now has broken so I have had to resort to using Git inside of the Linux Subsystem for Windows.
    • Though of course, I can still use Git inside PyCharm, my IDE of choice so far.

Now, about a year later I’ve been doing a lot more development on Mac and seem to have fixed the login issues. (I think I got a new version from Git and that was better.) I’m still using SourceTree because it’s easier to see the differences.)

MySQL on Mac

  • The Windows installer included everything. The Mac installer separates out the Workbench from the Server and Utilities. I first installed the Workbench cause I that’s all I needed initially, then I wanted to use the database and of course the database server itself wasn’t installed so I had to instal that separately.
  • After that I still had problems getting it working on the command line. The installer didn’t add it to $PATH – I had to look up what was wrong, and how to fix it manually. Following instructions on the internet, I had to add the symlink /usr/local/mysql/bin to $PATH
  • Then have to reset the default password
  • ALTER USER ‘root’@’localhost’ IDENTIFIED BY [password];

Now, just now, when I tried to run queries in MySQL Workbench, the application wouldn’t even run a simple query. When I used Google to search for [mysql workbench crashes on query], the first result directed me to install the later version, then I just used the built-in Check For Updates and got the most recent update. This fixed my problem too, now it works fine again.

Python Database stuff

Have to get the MySQL drivers for Python from here.

After much wailing and gnashing of teeth (well, a bit anyway) I just decided to use PyMySQL with is a pure Python MySQL module – apparently, it’s slower than using a C library, but it doesn’t have dependencies and just works. I had many issues getting MySQLdb or the fork, MySQL client to work on either Windows or Mac. For the size of my database, I’m not too worried about the speed of the implementation, particularly given that the main bottleneck will be the web scraping anyway.
Originally used the built-in Python HTML scraping library, but quickly changed to using Beautifulsoup. It is much more flexible and easier to write.

Of course, I actually got much further in the project as you can read about in the earlier posts, and also I probably shouldn’t let blogs like this go too long I suppose. It’s much easier to blog about the successes rather than the issues. If you have any ideas to make this easier… please let me know 🙂

Posted by Anthony in Course Chooser

Finished Another Course!

The end of this semester I finished COMS7201 – Computer Networks 1.  For some reason, it was one of the most satisfying courses I’ve taken.

I’ve written about all about it right here – Computer Networks 1, describing the assessment and several of the main concepts taught. Since writing that page, I still haven’t gone back and fixed the issues in the assignments, particularly Assignment 3. I want to implement the 1s Complement checksum and totally native IP header generation.  I’ve been doing other things with my spare time instead, like adding a feature to my Chat Log Parser/Viewer to group chats by day, writing for Weekend Notes on Mt Gambier plus other life skills updating and of course general life.

One approach I’ve started a few weeks ago is to create a daily goal to practice coding (and I’ve added writing) every day, but only for 15 minutes each day. This forces me to break up the task into discrete units of work – one (or two) unit test/s, one (or 2) method/s or classes, one paragraph or just the outline. I’ve found this method to work well so far, I don’t get put off with a long session when I want to relax but I still make progress. If I find I’m going really well and get into the groove (Flow), I can certainly keep going, but if not I can stop.

The above approach doesn’t work so well with university study because the assignments have due dates. This is good for personal projects and hobbies though.

Posted by Anthony

Day Grouping in the Chat Log Parser!

Finally, I’ve added a feature to my chat log parser to group messages by day. It’s something I’ve wanted to be able to do for a while now. This took a bit of work to add a new data structure and update the HTML Jinja2 templates. But now the base of it is done and if I want, I can customise the styles.

Screenshot of Chat Log with day grouping

Oh hey Sunny!

Originally, I tried to do the day grouping inside the Jinja2 templates, but this proved rather difficult – I ended up spending hours in the library trying to work out nested loops inside a templating language. Eventually, I realised I should perform the day grouping on the data structure instead, so I just loop over each day, then created the minute bundles inside.

Technical Info!

Creating the day grouping in the ChatLog data structure turned out to be surprisingly easy:

Continue reading →

Posted by Anthony in chat log parser

Urban Dictionary Analysis Tool

A fun little exercise I’ve been doing is a statistical and language analysis tool to analyse Urban Dictionary.  The idea for the project came about when it was pointed out that my own name was on UD and I realised that many of the definitions were of a sexual nature or offering praise to the holder of the name. I suspect that people are adding definitions of either their own name, or their partners or relatives. I thought it would be fun to programmatically analyse the various definitions and group them by content, maybe also ranking the most popular keywords or other interesting statistics.

The finished (though I’ll add to it overtime) product is available here:

screenshot of Urban Stats tool.

Continue reading for some technical details….

Continue reading →

Posted by Anthony