My time working at Paul Ramsay Foundation (PRF) was a challenging but valuable experience for me. It has given me invaluable insight into how large, philanthropic organisations function, and how real change can be achieved with an effective 'systems' based strategy. Given an endowment of around $200 million AUD a year, PRF is in a unique place because it can work to solve macro level problems, similar to government, but without the political and bureaucratic constraints of government.

I also learned a lot about how large organisations such as PRF interact with their partners (in this case, smaller charities) to create change. Keeping track of deliverables from partners while not being overbearing or creating unnecessary bureaucracy is a difficult balance to strike.

My main aim was to produce a data analyser tool that will make it much more efficient for them to use large datasets to gain insights and inform decisions, while adhering to data privacy requirements.

I was working in their Sydney HQ, which is a stunning old church that was restored and transformed into the PRF head quarters. I am working as part of the Measurement Evaluation Research and Learning (MERL) team to

1. Develop AI technologies that can be used to improve workflows and insights from public and internal partner data,
2. Utilize these tools to actually perform analysis, partly to prototype and refine the tools, but also to extract interesting and useful insights in order to improve decision making.

For the first few weeks, I was mainly working on the Natural Language Data Analyser (NLDA) tool that I have had in the works for a while. NLDA is basically a desktop application that takes datasets that you provide, and performs some kind of analysis on them. Previously, it was only capable of performing the task in a single action - it had one shot to do everything or it would all fail.

In order to make it more useful, I changed it to be 'multi step' and added an initial planning phase. Essentially, the task you give the tool is broken down into discrete steps (which are either of type 'Process' or 'Analysis'), and the code for them is generated sequentially. This allows one step to see the output of the previous step before it is run, so the AI needs to do less 'thinking' about what the output of a previous action is. The other benefit is that if something goes wrong, the failure is only localized to that one step, so only that step needs to be regenerated, and there is more clarity about where the problem lies. This happens quite often (the error from executing the code is sent back to the AI in order for it to be fixed on the next generation). By the end of the first week, this had been completed, and I also developed a user interface for it so that it could be more easy to use. This can be seen below:

I finally applied the tool to some PRF data to test it and generate some very interesting insights. It managed to handle complex tasks successfully (after a few tries, and with some experimentation).

I got it to compare the distribution of charities applying for a PRF grant with the general population of charities in Australia to determine the level of self selection at the point of application. There was a significant amount of data cleaning/processing required for this task, due to how charity data is provided by the Australian government, but it managed to handle this.

It even performed a bootstrapped analysis of the data, to control for some of the other characteristics of the charities applying, which correlated with the variables of interest.

I also asked it to do a natural language processing task - categorising some free text data from a dataset of grant applications, and it performed extremely well (the categorisation was not done via the LLM, but by running a natural language categorisation model locally, so no data needed to be sent off device).

I spent the last week showing the work I had completed to the team, and demonstrating how this tool could be used by them and their partners to make significant productivity gains. It was encouraging to see that the team seemed keen to try it out, and I hope that it can make a difference for them.