Growing Culture

A pre-post edit: I am finishing writing this and editing with the events of Nov 3rd in mind, which were certainly a hugely negative approach to motivating our team that my interest is a bit shaken, or maybe I just have to be more cynical, or less?

The Center for Open Science is growing, and its culture is changing. It sometimes seems like a confusing jumble of all sorts of not quite conflicting interests, but order has been coming to our controlled chaos. In some ways it is awesome, in others it might just deaden things.

I write this as one of our developers is leaving us. He was the first developer hired by the center and laid quite a bit of the groundwork that we use day-to-day. He has his reasons for moving on, but it is still sad to see him go. His enthusiasm really solidified my interest in working for COS, it was serious development, as well as being a serious cause.

I started in February when we had about 30-ish full time staff and many fewer projects going on. There was quite a bit of nebulous thinking with some really nice features that were making the OSF much more of a usable product. There was quite a range of ways to do things, different ideas and lots of freedom to learn and do.

At the beginning of the summer we brought in 35+ interns, doubling our staff numbers and blowing away any possible workflow. Some interns were working on small projects and others were working on large “R&D” projects that may never get integrated into the OSF. Essentially chaos ensued that we are still trying to dig out from under.

We hired a Product Manager and she brought more Agile methods as well as Jira and other ideas. This is certainly still a work in progress but a better definition of how a feature or a fix gets defined, worked on, and merged is a huge improvement over the previous workflow.

Unfortunately with the recent experience and some burnout starting to rear its ugly head, especially with a negative motivation speech to seal the deal and thinly veiled diatribes against the method we certainly need a moment to collect ourselves and our thoughts.

There are two huge projects that are essentially due this month. The problem is with the approach to these projects that it was not managed in such a way that makes much sense.

• One project was given to an intern. Not that interns can’t do a good job, however, he can’t put in the time when it gets to the point where we need daily turnaround on code review response rather than weekly.
• This project is now up to 12k lines. By far the biggest chunk of code that has been “through” code review. Why wasn’t it put through in a more piecemeal manner as it was being written? Because it was an R&D project, and though it had people signing off on the proposal, no one was assigned to help the intern bring it up to speed until two weeks ago.
• Now they are trying to solve the crunch time by putting more devs on it. It may work, but really the fight was lost when it wasn’t converted from an R&D project earlier in the game with more code review done on top of that.
• Also the full time dev assigned is one of the three code reviewers, meaning that he hasn’t really been able to do code review for other smaller PRs coming through the system.

So I could go on about the negatives of how we aren’t experiencing management in a good way, but I think it is not malicious, just some line of communication was not very clear, and now we are rushing to meet a deadline that we have just learned about.

So my first suggestion is that if the R&D project is ever going to possibly be more than just a learning project then it needs to have definite code and functional milestones where code and functionality can be assessed and reviewed in order for the storm to be avoided when we find out just how close we are to a deadline, at least we will know exactly how the project is going.

We also need everybody doing code review. I know most people aren’t very good at it, but that is why we have more than one level of code review. But three people looking at code that nearly twelve of us produce means that they are always falling behind. Always playing catch-up.

If we have better incremental code reviews with better chunking of the tasks, then we get feedback on smaller bits of code that can be implemented for the next chunk. When 500+ lines of code are changed in a Pull Request, it starts getting troublesome for the more thorough code reviewer to trace down all the implications.

I was going to end this, before today, with an inspirational thought about how we all need to be one of a kind without breaking companies when we leave. But I would like to focus on the more practical. We have a much better, tighter approach than we had before, but we still have some major hurdles to overcome.

Wellington and Django

There are few things I like more than learning, though Jessie did make a splendid Beef Wellington this evening. Learning has much longer lasting effects than just a perfectly executed dinner that came from me joking that I wanted it for dinner. Okay, dinner was awesome, now to more serious matters.

Seeing as the Django project has been out for the last 10 years, it isn’t really anything new to those who have been developing websites with Python by any stretch of the imagination. For seven of those I have been working on telescopes, not websites, so I only knew of it tangentially when I started to explore Python.

After finishing my previous projects at work, after a big project that took a bit longer than expected, I was given an administration project, using an Admin interface that had been an R&D project for two of our interns over the summer. They are both still around, working part time, so it wouldn’t be nice just to tear up what they had been doing all this time, especially since they are still working on it.

I talked to a few people familiar with Django and through some experiments and more conversations came to the conclusion that adding the module as an app to the Flask project was probably the best idea at the moment. Though typing that it seems to be quite a silly thing to do. However, it is my intention to make the Django app as modular as it can be, so if we decide to rip all of our models out of the website and package them, we could in fact remove the admin app and make it a service, or a “micro-service” if you so prefer.

I just wanted to mention that we got a book for the office: Two Scoops of Django: Best Practices for Django 1.8. It is already a pretty good read and I am only a fifth of the way through the 480+ pages. The Greenfelds definitely deserve the ice cream that the cost of the book will buy.

Open Science, so what?

In reference to my job at COS, of course people ask me what I do. That is pretty easy to answer: I make websites work better for scientists to use and keep workflow/data. Great. But the question often not asked, maybe because adults left it back in their childhood and it might be considered rude: Why?

Why did I work for telescopes? Why is this current job more than just a pass-time in order to make money to support myself and my family in my old age? Two things, one intrinsically selfish, and another less so, but not completely altruistic: I need motivation in order to do things and I can go for ages without generating any motivation myself, and science needs to be more open.

So I need a bit of motivation. Once given a bit of outside motivation I can often generate enough extra to do a pretty good job at whatever I am focused on. For example summer semester AI course was definitely a motivator, but I liked it enough that I put in the effort to get an ‘A’. I am not certain how much I will like this next course, but proving to myself that I can do A-grade work means that I can convince myself to keep up the work. I will talk more about that course soon.

Anyway, enough preamble, really why am I motivated to do the work?

It comes down to the fact that I believe that we need science. Do I believe that science can solve all of our problems? In short: no. However, it can give us explanations for many questions. Those generally lead to more, or finer focused questions. But it’s the progress that has been made previously that colors my optimism for what may yet be discovered.

The problem is that today’s professional scientist/researcher falls into a system that has the catchy phrase of “publish or perish.” Not a phrase that inspires confidence in those unable to find something publishable. And that is just it, what isn’t published? And why isn’t it published? I am not talking about pseudo-science with obvious flaws and downright false conclusions, but rather negative results.

Negative results come about when a scientist poses a hypothesis, runs experiments, and then gets results that say the hypothesis is wrong. What’s a scientist to do? There are many options, the first of which is the least insidious, but still not good: changing the hypothesis to fit the outcome. But if negative results aren’t published, how can we avoid duplication of effort? Not to say that checking results is a bad thing, in fact being able to reproduce results in another study is great.

Let’s imagine for a moment that there existed a society where parents only told their children positive outcomes, only things (true or false) that could help them. One of the first things you notice is the consistent burn scars on the hands of just about everyone that you meet. The first few people that you meet won’t talk about the scars and the way the react the question seems to be taken as a rude question.

Finally one older guy chuckles and shakes his head when you ask the question again, carefully trying to choose the right words. He replies, “We are a positive society, we don’t tell people about bad outcomes. Everyone has to discover that the stove is hot for themselves, and some are more badly burned than others.”

Or to put it into current day science culture terms: What publisher is going to publish my negative result so others can avoid doing an experiment that will also have a negative result?

Maybe other scientists would be able to move on and try a different angle, or derived questions from the negative result?

But it is more than just publishing negative results: In order for science to pull itself, and its associated parts, out of the decline in popularity it needs to have a vitality and veracity that doesn’t allow otherwise uninterested people to ignore it, or disparage science in general.

Why I like working with and for science is because there is room for improvement. It isn’t just about catering to the masses, but rather discovering things for the masses, and letting them know about discoveries in a transparent way.

There is negative science, but there is also science results that are wrong. When data is falsified or manipulated in such a way that results are completely twisted around, then this is bad science, not just negative results, but false results. If we can help weed these out by making science more reproducible, then we are well on our way to making science trustworthy again.

I am optimistic that I can help in this area by making better tools. That is what motivates me day to day.

Contrasting Old and New

What’s the best thing about living on the mainland after seven years of living on Hawaii? Flights aren’t working out in my favor as far as schedule and cost, nor is it cheap to send furniture from my parents’. In Hawaii? Suck it up, the only way you are getting off this island is to fly or get really good at sailing. Also freight is going to cost you 1400 bucks to ship an end table… okay maybe not that much. Mainland? Okay, we will drive 3200 miles round trip for about 500 bucks, be able to take all the luggage we want, and then turn around and bring home the furniture. The bad news? We are looking at doing that in December and January. Oh seasons.

Whatever plans we have though, I think both of can be pretty flexible and be able to go before any large storms are going to occur. Then we won’t need to be as regimented to catch flights. And if we really need to we can go south. Add a few hours or more, but still make it in our own time. Also, if the weather is bad on the days we have plane tickets, the changing of plans or waiting would be frustrating to say the least.

What car are we going to drive there and pick up furniture with? Ah, but we have a direct line on a Subaru Forester, buying it Saturday. It might be the car’s first real trip period, but we are really excited to get it.

Furniture, but not really -

Due to office remodeling I also got cabinets with a countertop,  going to install that in the garage for a workbench. Might be modifications throughout the years to make it more useful, but I think it will be quite a bit easier than trying to build my own with basically no tools currently.

Short observations on work

One of the best things about working for a startup is the fact that there is huge right of way for creativity, in order for an idea to be really crazy it has to be at least one more level of crazy than normal. To put it into perspective, we have 30+ interns working for us this summer, about half the company. That is nuts in my mind, but the amount of work they are doing is actually amazing, in fact swamping our normal processes. But speaking of processes, still a large amount of room for growth or massive change.

Compared to the Joint Astronomy Centre the Center for Open Science is a massively younger, more flexible place. This is both a blessing, but it also means that we aren’t playing with 120 ton telescopes that can look at a gas cloud being eating by a black hole in the middle of our galaxy, trade-offs.

Probably my biggest issue is that there is so much going on that it is hard to keep track of who is working on similar or potentially conflicting things. I have to say that one of my merges of code essentially wiped out an entire commit by an intern, not because they did anything wrong, but because I started using schema validation rather than writing out a dataset line by line. Also there have been a couple cases where interns figure out that an issue already has a PR that just isn’t merged yet, after they code a solution. Even with all of that and the semi-chaotic nature of interactions I have to say I enjoy it nine out ten or fifteen days which is a bit better than some spans of time at the JAC.

And I have certainly learned a huge amount. That is not all that different from the JAC, but the pace is much faster, and the tools are so much newer in general at COS that sometimes you have to fix them and send in a PR to a stand alone tool. But that leads to other things: Open source! Oh man, so glad to still be working for science and for the greater good. I am not sure if I could work for a “business” company, even if I’ve dreamed of starting my own sometimes.

Anyway, on to solving the world’s problems by facilitating a better website for scientists to do better, more transparent work.