Wednesday, December 05, 2018

Fair Warning

If you ever want to enjoy a movie, radio show, or anything with bird sounds in the background, don’t ever learn bird calls and bird songs. I’ve said this before, either in discussion or maybe even somewhere on this blog. Actually, in counterpoint, when they get it right it really helps set the scene. If you want to start learning bird songs then try this Bird Academy class. Or if you are already somewhat experienced you could try the Quiz.

This quiz that I speak of is actually the project I have been working on the last three months. It is a quiz consisting of mostly unrated media, photo or audio, from the Macaulay Library archives, most of it contributed through ebird. I know it’s sort of like my child, but I absolutely adore it. I already took enough quizzes that I was mostly familiar with 80% of the birds in Japan in May…

I really would like to jump into the spaced repetition, but if as one of the rules in the link says you have to make your own cards, then it would be really hard for us to do with our random species and asset selection. It isn’t totally random, you can pick a place and a time of year. We get a list of likely birds seen in that part of the world, you will be disappointed with Barrow, Alaska in December, and then pick 20 random species. Then we get the photo or audio clip with the least number of ratings and serve up the quiz.

This works well for the most part for well inoculated users, there are terrible tiny photos and misidentified photos, but in general I really can’t get enough of it myself. There are a few issues that I think would make this a better experience for someone trying to use this to learn a new region: Weight the chance to see a species by the number of times you have seen it previously, and try to never show the same photo to the same user if it can be helped.

Let’s say that you are trying to learn birds of the Northeast US. One of them is most certainly the American Robin. At the moment in my area there are getting to be around 200 potential species. If you take the quiz 10 times and see 200 photos then you might have seen the good ol’ Robin about five or six times, and other species on an average of two or three.

This is a several runs of completely random choosing of species for quizzes. At 10 quizzes the user will only have seen 125 species of the 200. Even at 50 quizzes there are still 4 species the user has seen once, though because of long tails and randomness 40 quizzes should be a better indication at 13 species seen less than twice, or rather no chance to learn the species.

So totally random is not really a great way to go. On the other hand totally not random is an algorithmic nightmare unless handed over to Machine Learning or better: Machine Teaching. But let’s say we need a bit of middle ground. How about just calculating a weighted random selection that favors the species a user hasn’t seen yet? Basically every time the user sees a species, they are a bit less likely to see it. It isn’t a perfect solution, but it would be that much better.

This second approach shows a much steeper decline in the number of unseen species. The tail is still potentially long, but we have a much better chance of seeing more species over the course of fewer quizzes. We are only missing repetition for 57 species after 10 quizzes rather than 75 for the completely random approach. I know this doesn’t sound like much, but 20 species could be significant, especially if they are the most common species.

The second issue is a bit more tame. Essentially we are sorting the possible assets by how many ratings they have and then taking the most recent one with the least ratings. All we really need to do is get rid of the most recent and check for a user’s previous rating on a piece of media. It would take more time, but it could make it that much more rich. Really this is only a problem for species that have very few photos, so go take more! Maybe also unpopular bird species don’t get enough attention in that there isn’t enough turnover to churn-up the number of ratings on a regular basis. Hopefully will see less of the latter with more participants.

Why am I geeking out about this? Well it goes back to when my wife was studying birds for her two jobs doing bird surveys. She hand-built study decks and then reviewed them every night with Anki. Of course that fits in with the spaced repetition from before, but then I started working at the lab and saw how many photos we had, what a treasure trove to build something that could build a set of flashcards from 26k+ Robins, etc.! Starting this was like a dream come true, and through some not-so-dreamy work here we are. It needs some tweaking, but it is still superb.


I wrote this back in March before we released the app. The whole idea of the app is to help Macaulay Library to curate data via community ratings. Of course it is pretty close to being a learning app that people can learn birds with, but ML doesn’t want to step on toes. Nine months in we have collected 3 million ratings which helps us to show some of the best photos the community has taken. Ratings were critical in the release of eBird species pages where 6k previously un-curated species had photos and audio chosen if they had any.

I think the biggest wish now is to add a bit of filtering. For instance people might be well-versed in their finches, but want to focus on their shorebirds, even if you are birding locally. Currently with hundreds of species it could take quite a few quizzes to learn all of your shorebirds, though we would get quite a few ratings in the process. It is certainly a possibility, but we have plenty of projects to keep us busy, even for a good part of next year.

I've taken a few more since the start.

Tuesday, December 04, 2018

Conflicted Progress

Read these two articles:

How can these two exist at the same time? One must cancel out the other like antimatter/matter annihilation. Or can they both be right?

Interestingly, I think both are possible, though the “Superpowers” article is mainly writing from the viewpoint of survivorship bias. That is the author is saying that because they did it, so can you, and because you are reading it then they succeeded wildly by following their own rules.

The Farnam Street Blog looks at several cases of Winner Takes All, but doesn’t tell you how to overcome that, except maybe buying several thousand copies of your own book. Does it cancel out the rule-following concept in the more positive article? No, but I think it tempers it quite a bit.

When it comes down to it: We can all improve, maybe even the top performers, but really the only person that would have thought we have superpowers is our former self. Let me frame it with the Recency Effect.

Let’s say that you are a hotdog eating contestant and you can eat ten hotdogs in a minute. You are so good that you have sponsorships and tour the country. Then someone comes along and introduces a new technique and is able to eat one hundred hotdogs in a minute. At the behest of your sponsor you learn the technique and are able to eat one hundred plus one hotdogs, reclaiming your crown.

Being at the top and then reclaiming the top certainly looks like superpowers to folks lower in the pecking order, but even the lightweight packing away only eighty per contest would scoff at your old record of ten due to your new achievement. You might appreciate the difference, but you only really have your initial reaction to gauge just how monumental a jump that is.

All’s good in hotdog-land, but what about starting from the bottom like removing a bad habit? Initially people may notice if it is an obvious habit, but soon it will be old history. Good changes are like a differential for people noticing and hopefully a integral in improving life. That is: people will only really notice as the change is happening, or if asymptotic like a haircut, right after, but then that will be the new normal. But maintaining a good habit like brushing your teeth might literally improve your life expectancy, and no one runs away from conversations anymore.

Self-set rules will only get you so far in becoming the best, the rest is how they relate to your goals, but I think they are a huge step in getting you away from the worst. Rule-following has another name: discipline.

Monday, December 03, 2018

And It's December

I just turned in my final assignment. By final I mean final assignment, meaning the last one in my master’s degree. I mean like if my team doesn’t miss more than four out of 16 points I will graduate on the 15th. I mean final like there is a chance for extra credit that is bigger than any other assignment that might mean the difference between me graduating and not. I dislike this class with a passion.

The degree, Master’s of Computer Science at Georgia Tech. The final class, Introduction to Health Informatics. Would I ever suggest taking this course? No, just a straight no. The only reason I can see for someone to take it would be if their job is going to deal with Health Informatics. If they are already dealing with it, forget it, they will be happier not taking this class.

For those of us that were never going to go into healthcare software this is really just an exercise in frustration. The paramount frustration is supplied by the course itself, or rather the original instructor. He essentially describes all the different ways everyone is doing healthcare data, and what a nightmare interoperability is. The tone is set in frustration that even after 60+ years of thinking about electronic health records that it’s still a really hard problem. Just like telling doctors they aren’t doing something right.

The second frustration was how the class was run. It seems that it was mostly TA run because the professors aren’t actually technical, but having taken classes by David Joyner I found their expectations did not normally match their directions and the grade weights for particular sections that were administrative details far exceeded actual content. Missing a name in a presentation was the difference between getting 100% and getting a B in one case.

The third frustration was with our TA, responding to pertinent questions after the assignment was due. Nothing better to make an assignment even tougher.

If there was an alternative, I would have taken it. And throughout the 4 years that I have labored through all of these courses there have been a few that have been promised to whole time, one that would have been interesting is the computational journalism or something like that. It couldn’t have been worse than this one, right?

In some ways I really shouldn’t complain, the main one being that there were no tests. A few courses with tests have been my worst courses. Give me coding and projects over tests, I am just not built for them anymore, who is with google right there?

The amount of time and money that I will have available in the future is sort of daunting. What will I do with all that time? But I think saving the money is going to be pretty obvious for that. I guess I can rewrite my novel, write another, and spend time with my wife. She’ll like that.

Wednesday, August 22, 2018

Last Class!

One last class! I am a bit apprehensive, but since a large chunk is a group project and I already have seven group members I think that we will be able to do okay. I am not entirely sure what role I should take in the group, everyone responded to my post so it feels like I am coordinating, but I don't necessarily want to be the Project Manager.

I think I see everyone as having two roles, one of PM, Quality Assurance, Video Editor (for our presentations), and then everyone is a developer with a range of different modules/tasks. I think we can start talking about this and maybe formalizing our approach for the next month before we get or choose a project.

It is unlikely that I will post much during this period, hopefully next time I will have a Master's of Science in Computer Science.

Wednesday, March 14, 2018


I will not complain about my proposal being failed just to get my attention, I will not. I certainly need to fix it even though it is better than most of the ones that I read and got some useless feedback about how it was better than the reviewers’.

Alright, but I do need to complain, you know, righteous indignation and all of that. This week I received a credit card promotion from my own bank. Probably because I still have a cash-back card and they are tired of giving me barely a penny on a dollar. They want to transition me to a card that is “better.” Great, but that is only a small grievance, the real sticker is the copy text at the top of the promotion: “Use your Bank-name-here credit card to help pay down debt.”

Um, no. A credit card is not used to pay down debt. A credit card is used to accumulate debt.

Let’s get biblical for a second, debt or assurance of debt is a chain around your neck that can be yanked by the person that is owed. Go check out Financial Peace University. Although we don’t ascribe to the entire system described by FPU it is certainly closer to what we practice than not. Just average credit card debt is a bit disturbing. Maybe we should look at getting rid of it. With only a house to pay for at the moment it is sort of silly to have a monthly accumulation as well. Unfortunately for the simplicity of not racking up debt to be paid off monthly debit cards are not as well protected as credit cards, hmmm…

I started and finished Ready Player One by Ernest Cline so here’s a review. Overall: Not a terrible story, but talk about IP nightmare for the upcoming movie. I am a geek, nerd, whatever, but the level of 80s pop-culture was way beyond my ability to cope, even recognizing 60-80% of it. But here is where it is ultimate:

The disconnectedness from the real world was certainly surreal, but then was conveniently forgotten at the end. There are so many points in the story where it could be an interesting delve into futurism or even keeping self-consistent and then it just throws in an 80s reference or several pages of exposition and then brings us back to the moment reeling. Some moments stand out in my mind, but maybe because they are so few. On top of that all the victories are essentially hollow, there is existential crisis throughout, but it doesn’t seem to affect any performance on challenges.

Would I suggest you read it? Was it a waste of time? Meh on both counts. If you are a super fan of the 80s then this book was written for you, but it wasn’t even a waste for me because I can see what not to do in my writing.

How about that Black Panther movie? It was pretty good, I wouldn’t jump straight to the best in the Marvel universe, but only because it is really hard to compare all of them side-by-side. That said, it did talk about some issues that translucent-skinned individuals like me don’t often think about. Well done, well told, worth seeing.

Saturday, February 24, 2018

How to Compound a Rainy Day

Watch the solar output when it’s super dark and rainy; it’s like watching your investments tank on a bad market day, but with even less ability to control it. Yeah, but it certainly compounds a sunny day, or even better: mitigates having to stay inside during an incredible day. Okay, barely.

This week I dove into the world of not Numpy with Java, ND4J. I didn’t even get into the ‘N’ of ND4J just keeping to a one dimensional array. Let’s just say that it has many really nice options for dealing with arrays. I will not mention the fact that you cannot easily get an array back to an otherwise usable list in Java without some machinations.

I think that is what I miss most about python in general. Even if a library didn’t have something very specific for a certain community it’s likely the community had someone willing to dive in and make a module that can do it and then posted it. Java certainly has its libraries and all the people using it, but somehow it’s a different community.

When someone asks how to do something there are ten different answers and they must all be encapsulated with a class. Sure if I want to do this again I will move it to a function. If it becomes widespread and is more like an object then I might think about making it a class, but seriously. And then spring… I just want to curl up with some nice classy class based views in Django and leave the mess of Spring behind.

In other news, I ran twice this week. Tuesday and Wednesday were up in the 60s and this morning wasn’t too chilly at 40. If New York only has 2 months of real winter, and it was cold, I guess it’s better for my running. Cloudy, but keeping at around 60% of electricity used produced by solar, ran, and now off to clean something and write my class paper proposal. Content.

Tuesday, February 13, 2018

The Only Odd One is Me

Not to stir up any fears, but here we go. That in-the-moment feeling of everyone watching, while everyone else says they are not. Can we be real for a second? If you are not staring at a screen in a crowded place, how observant are you? Do you notice when someone does something, at least innocuous such as tripping on flat ground and then looks around to see if anyone saw? Yeah, sorry, I saw that one. Shall we just share a smile and keep going forward?

The problem is that there are so many non-innocuous situations that anything from dumb to malevolent. Going to this World Life Expectancy site can be a bit eye opening, plenty of malevolence, but plenty of stuff that I could classify as dumb, but really it is just sad. Not sad as in “those people should have seen it coming,” rather maybe it could have been prevented by better/earlier treatment or just better friends. Maybe in my age group the traffic accidents could be helped by smarter cars

Anyway, there is a point to this: Whatever I am suffering that is not acute AND terminal, there are possibly a few thousand people suffering of the same thing. That is true of post-graduate online masters degrees. Of artfully procrastinating by practicing writing. Of making bad lists with weird sentences. Though that last might be just me at the moment. But there are people suffering, or...  Oh, how odd am I?

I am odd, there are so many things that I am bringing to the front of my mind through biases where I haven’t really met many people that do, or have done, a permutation of different things that I have. This is the deal with permutations, even if someone else has done everything I have, but flipped the order of two events, then by definition we are not the same, because we experienced those two events with different views. Twins can be a prime example.

You are odd as well. I have found out, somewhat too recently, that I have no idea what is going on inside your skull and trying to understand every nuance is impossible and/or maddening. I spend so much time with my wife, yet it is better to have a model than to be “sure” of what she is thinking.

What frustrates me more is how odd I feel in trying to make people understand my thoughts. I think my thoughts are pretty good, the problem lies where your thoughts conflict with mine. Either I am too dismissive because I am too absorbed by my thoughts, or I have already considered it thoroughly and have a ready answer. Though that seems dismissive too because I cannot explain it well enough.

Four years of theater should have made me a better actor, if not in hiding what I think, at least in having the poise to articulate it better. Well no, I get cut-off when I need a moment to speak, but then ramble when someone should just cut me off. Can I have it both ways? Thinking, listening, and responding. How might I even practice that? How can I even find the place to practice that?

I would love to tell stories too, but I would for now settle with schooling on conversation. I have to say that I write better than I did back in this post, the first, from twelve years ago. Even so, I can improve my writing. Better to practice than to pine about not having practiced. Only a recent addition, that, maybe it will stick.

No one is odd in that everyone is odd. Maybe I will find some more odd people that we can be mostly mutually accepting of one another.

Featured Post


John studied himself in the mirror as best he could through tears. Red, puffy eyes stared back at him, a running nose already leaked just a ...