Hellonline (Eran’s blog)

Just finished reading On and Off the ‘Net: Scales for Social Capital in an Online Era by Dmitri Williams. As the title suggets, the article describes a new measure for Social Capital online and offline. An interesting read despite being a little technical (not “my kind” of technical, rather statistics and social science technical). Looking over the list of questions for measuring social capital online and/or offline I found myself thinking that I can’t really answer those questions properly anymore. It’s not that I’m lacking in a social life, quite the opposite actually, but my online and offline lives have almost completely merged (and in some cases flipped) since I moved to San Francisco.

When I lived in Israel my online life consisted of friends in the US that I connected to via IRC, Email, IM, etc. and offline friends and family that I saw and interacted with in person on a regular basis. Now those two have flipped and some of the people I was closest to in RL became online-only entities (except for when I visit back home). That’s interesting but somewhat expected considering that I flipped my life around by moving half way around the world. What’s more interesting to me is the merging of my online and offline life here in SF.

I have two main clusters in my social network. One is a group of friends that mostly formed online on tribe.net and then became a strongly connected real-life urban tribe. The other is a group I met initially in real-life (if you can call Web 2.0 parties real, that is) and I now experience on a daily basis via twitter, blogs, IM, etc. I am still a part of both groups both online and offline which makes separating my online and offline lives pretty hard. @rk and Ryan are the same person and the same is true for almost everyone else I know; there’s nowhere to draw a line.

It might be just me; I am, after all, mostly an introvert and I don’t meet new people very easily. I’ve pretty much saturated my social capacity so I don’t participate too much in online or offline activities where I can form new weak-ties (aka networking) also my friends and I are mostly very comfortable with technology and use it to communicate constantly. But I cannot help but think about the disconnect described in the beginning of the paper describing social researchers who failed to recognize that the Internet can be used for (and is indeed a hotbed of) social activity and see how a similar shift might be occurring now.

To me, the Internet is just another way to communicate. In many ways it’s a more efficient way to organize my social life, coordinate with friends and keep tabs on what’s going on in around my social circle. Separating my online life from my offline life is a futile effort and just doesn’t feel right. It’s all just Life and it’s all Real.

Now that everyone’s gone to SXSWi and left me here, alone in the Bay Area with nothing to do and all this parking (not really, there’s still no parking) I finally have some time to be productive. To make sure that my twitter stream remains somewhat useful for the next 10 days or so I created this small Greasemonkey script to filter out all that South By Noise(tm). It might not be useful if you use twitter from some desktop client or IM but if you’re like me and you’re still using Firefox this may actually be useful.

Notes:

• Yes, you need Greasemonkey to run this.
• Yes, it probably has bugs.
• Yes, it may cause your computer to implode and your entire online identity to be sucked into the ensuing vortex. Use at own risk!
• Last, I based the script on the Reddit Content Filter script by pabs. So, thanks dude!

Oh, install the script and add banned words and authors using Tools > Greasemonkey > User Script Commands > Edit Blocked (Users/Keywords)…

Update: Slight bug fix. Latest version is 0.21

Note: I am by no means an expert on Java Garbage collection. What follows is a description of my attempts to battle memory and gc related issue in our server. If you think I’ve been especially thick about something, missed some obvious options or went completely off-track, please let me know. Also a warning, this one’s kinda long :)

During our latest bout of long-running, high-load endurance tests we’ve identified Garbage Collection as a main obstacle. The pattern observed after about 2-3 days of running at high load was very frequent full garbage collection cycles, each causing a full stop of the JVM for several seconds. This of course led to very slow response times from the affected servers. Over the last week we set out to improve the gc behavior in our application.

The Application

• A highly concurrent web application running on tomcat 5.5 and Java 5.0 on multiple app servers.
• Using Hibernate to access a MySQL database with a distributed EhCache as 2nd level cache.
• Most requests are very short lived, less than 100ms response time.
• Some background processing, in-memory caching of frequently referenced objects and some shared objects pools.
• Running on dedicated dual core servers with 8GB RAM.

Trials, Tribulations and Garbage Collection.

Short response times are important so we’ve been using the low-pause garbage collector (also known as Concurrent Mark & Sweep or CMS) quite efficiently for shorter runs (around 2 days). Things started falling apart after a longer run that did not employ second level caching. My assumption at the time was that CMS is not particularly efficient at this sort of situation (a large number of short lived objects and a small number of longer lived ones) and so my first step was to re-enable second level caching.

With the cache enabled we did see an improvement in application behavior but there was still very clear degradation in behavior. Most notably the amount of memory freed in Old at every cycle of tenured generation GC was getting smaller and smaller and cycles were getting closer together. CMS does most of it’s work concurrently except for two point during which all mutator threads are paused while the GC marks all live objects. Seeing these cycles happen closer and closer together meant that more and more time was lost to GC and more requests were likely to be delayed for 3-5 seconds while GC was taking over. My first goal then was to figure out why those tenured generation collections were happening so often.

Fortunately the EhCache page about Garbage Collection contained some useful information: Use XX:+DisableExplicitGC to disable any libraries calling system.gc() and forcing full GC cycles. There is a known problem when using RMI in java (pre 6.0 at least) where RMI causes full garbage collection every minute by default. To remedy that set the following parameters to larger values (both are in ms and default to 60000 or 1 minute)

• -Dsun.rmi.dgc.client.gcInterval
• -Dsun.rmi.dgc.server.gcInterval

These changes made a noticeable improvement but tenured GC cycles were still happening more and more often. I started digging around some more and learned that CMS starts to collect the tenured GC whenever the Old generation gets to be around 65-68% full. Now things started to make sense. With our GC becoming less efficient over time (possibly because of a memory leak, possibly not), the occupancy rate of Old was creeping up over time and so it was hitting that 65% threshold more and more often causing GC cycles more and more often.

Now that I had a better understanding of what was going on I could started looking at some of our GC data (collected using -verbose:gc, -XX:+PrintGCDetails, -XX:+PrintGCTimeStamps and jstat -gccause). The first thing that struck me was that the size of our new generation was way too small for our requirements. It appears that the default values for CMS are heavily weighted toward the Old generation. I started experimenting with -XX:NewSize and -XX:MaxNewSize to explicitly allocate more heap to the New generation. I also started setting the total heap size larger.

We started at 3GB total heap (on 8GB RAM machines running nothing else but tomcat) and things seems to run much better at 6GB total heap and 1.5GB allocated to New. At the very least the additional memory bought us some time before we hit the really Bad Times(tm).

I now started looking at slowing down the rate at which the Old generation was filling up. My initial thought was to try and get less objects to tenure at each Young Generation collection. To do that I increased the size of the survivor space (-XX:SurvivorRatio=8 to allocate 10% of New to each of the survivor spaces) and started tracking the age distribution at which objects in New got tenured using +PrintTenuringDistribution. After some toying around with the parameters I got to see some objects live up to age 2 or 3 (initially the maximum age I could see was 1) but I still could not see much of a change in the rate of increase in Old.

We eventually made some sense out of it based on the argument that most of our objects are either very short lived (1 transaction length, around 100ms) and would get reaped right away or very long lived (either pooled or cached) and therefore should be tenured as soon as possible. Based on that reasoning and the VOIP related case study outlined here we’ve settled on immediate tenuring of all objects that survive one Young Generation GC cycle. To achieve that we’ve set -XX:MaxTenuringThreshold=0 and -XX:SurvivorRatio=1024 (the later just to save some memory as survivor space will no longer be used at all).

At this point things settled down a bit and I turned to some fine tuning. Playing around with the size of New space it seemed that we get a good balance of GC frequency and pause length with about 1024m dedicated to new. This kept Young Generation GC frequency to around 4-5s and Tenured Generation between 1.5 and 4 minutes with about 3-5s pauses depending on load.

We made a couple more attempts to fight the slow increase in the size of Old.

• -XX:+UseCMSCompactAtFullCollection to force compaction of memory at GC time based on the assumption that we may be losing available memory to fragmentation (fragmentation is a known problem with CMS). After running with that setting for a while it seemed to make absolutely no difference.
• Reducing load on the servers for an extended period of time. We wanted to see if we were encountering some inefficiency in Java’s GC (at least with the parameters we have and under high load). After increasing the load back to initial we saw the usage levels of the Old generation go almost immediately back to where they were before the “rest period” and continue to increase in the same rate.

Based on those two experiments our current assumption is that this slow creep is due to a memory leak. We’ll have to confirm that by some deeper profiling of the application.

Related Parameters and Tools

• -XX:NewSize and -XX:MaxNewSize – Explicitly set the total size of the New Generation (Eden + 2 survivor spaces).
• -XX:SurvivorRatio – Set the ratio between the size of Eden and Survivor.
• -XX:MaxTenuringThreshold – Set the maximum number of times objects will move between Eden and survivor space before being tenured.
• -XX:+DisableExplicitGC – Disable explicit calls to system.gc() which may be hidden away in some library.
• -Dsun.rmi.dgc.client.gcInterval and -Dsun.rmi.dgc.server.gcInterval set the interval between full GC cycles caused by RMI.
• -verbose:gc, -XX:+PrintGCDetails, -XX:+PrintGCTimeStamps and +PrintTenuringDistribution – Useful for tracking GC behavior in your application.
• Jstat – when used with -gccause or -gc is another great way to track GC behavior. My favorite command line for jstat:
• jstat -gccause -h10 <pid> 1s

Useful resources

• Tuning Grabage Collection with the 5.0 HotSpot JVM – Contains a good explanation of the main GC algorithms in the JVM and most of the relevant parameters.
• Sun’s GC FAQ in HotSport JVM – answers many questions and discusses parameters barely mentioned elsewhere.
• Garbage Collection recommendations for EhCache - If you’re using EhCache, this is definitely a good place to start.
• Enabling Java-based VoIP backend platforms through JVM performance tuning – Contains some real world data and recommendations. Also good discussion of some of the more obscure options.
• Tuning Garbage Collection in the HotSpot JVM (from IBM) – Good information on tracking GC performance. Gave me the push I needed to actually start graphing this data.
• Last but not least, Jon Masamitsu’s blog – Deep information from inside the belly of the beast.

At least not in the long run. In most cases. Well… here’s a glimmer of hope anyway.

Earlier today, Facebook sent warning to developers whose apps forced users to invite friends in order to use the app, turning off invitations altogether for those apps until that behavior was corrected. This was a major user experience problem that now should exist no more.

I’ve been having this back and forth discussion lately about Send Hotness just one of many Facebook apps that force you to invite X many friends before you get to use it. I wasn’t surprised (although somewhat disappointed) to see the app take off in a viral storm and reach hundreds of thousands of sign-ups or more. Despite their apparent success I maintained that in the long run this kind of behavior will not pay off. Irritating your users is not good policy especially if you’re trying to build an enduring brand. I’m glad to see Facebook cracking down on overly-aggressive applications. In the end this kind of policy will prove beneficial to everyone involved.

Here’s to less SPAM!

We’ve all been saying this to the music and movie industries for quite a while. More DRM does not translate to more sales, Innovation does. It seems that the gaming industry is now learning a similar lesson:

As we believe that we are decreasing the number of pirates downloading the game with our DRM fixes, combining the increased sales number together with the decreased downloads, we find 1 additional sale for every 1,000 less pirated downloads. Put another way, for every 1,000 pirated copies we eliminated, we created 1 additional sale.

Though many of the pirates may be simply shifting to another source of games for their illegal activities, the number is nonetheless striking and poignant. The sales to download ratio found on Reflexive implies that a pirated copy is more similar to the loss of a download (a poorly converting one!) than the loss of a sale.

It’s good to see another company learning that the standard rhetoric about how piracy equals lost revenue is almost completely false. Most people who pirate your product would probably never have bought it in the first place. So why ruin your paying users’ experience by more limitations? Instead innovate.

In Korea game piracy has long been a problem. Gaming companies in that market turned mainly to two other business models:

1. Subscription. Works great for WoW! Moves the authentication and authorization work over to the server and makes pirating your content much harder. Also it gives you a steady revenue with which to keep developing new content, getting your users to stick around longer, pay you more money, etc. Unfortunately for the Korean game market this method did not seem to work very well over there which is many have switch to this second model
2. Free to play games with micro-transactions. Download the game for free, send it to your friends, install it on as many machines as you’d like then come check out our in-game market, it has some nice power-boosts and unique items that you can’t get anywhere else. This model has been fantastically successful in Korea and is now slowly infiltrating US markets as well.

What’s next? It seems like casual gaming inside social networks (read: facebook) is set to grow but with a so far unclear business model (ads? microtransactions?) That’s fun but here’s what I’m waiting for: with Guitar Hero and RockBand sales going through the roof, I’d love to see the next music CD that I buy come with a bonus Guitar Hero or RockBand track so I can Rawk Out with my friends to some cool new music. That’s the kind of innovation that would actually get me to buy CDs again.

A few days ago I came across a couple of interesting articles about creativity. Both are somewhat focused on design and user experience but I think they’re worth reading even if you’re not in those fields (I know I’m not).

Both posts were written by David Armano of Logic+ Emotion. The first post, Are You a Synthesizer talks about those people who can look through the noise and see the pattern. The second was linked from that first one, an article on UX Magazine called Creativity 2.E talks about the evolution of creativity today. Pretty good stuff.

After a long forced hiatus, HellOnline is back! I finally own my own domain so no more annoying DNS related outages (until the next annoying DNS related outage at least).

Some updates:

• I’ve been working at DigitalChocolate for just over a year now. I’ve been working on several projects some of which you can check out (like Dchoc Cafe or the Facebook version of Towerbloxx) and some others that are not quite ready.
• I’ve been doing a lot of load testing and performance optimization lately so expect to see some gripes about MySql and Java garbage collection.
• I’m planning to get started on the next version of WhereAreYouCamping.com soon so if you’ve got any ideas or requests, please send’em my way.

For now, I’m off to celebrate this miraculous resurrection. Soon, a new design and new posts.

In the olden days (I’m told) there used to be this attitude among some software companies that said “we don’t do QA, users do QA!�? after a generation or two of software engineers we’re now safely in the hands of test driven development and all is well with the world. We’ve moved away from the pioneer, get-it-done age to the well-established, get-it-done-well age. I believe something similar might be happening in the world of blogging.

Quoth Michael Arrington: My readers do the final fact check for me. And if I’m wrong, I correct immediately. Sound familiar? If your readers know more about what you write than you, why is it YOU that’s writing? Let them write it. I don’t take on software projects I can’t finish and you shouldn’t write about things you don’t know enough about. For example, I’m still waiting for a complete rewrite on the twitter vs. dodgeball post which was written by someone who obviously has close to zero experience with either dodgeball or twitter.

No newspaper would let that article pass same as (almost) no software house would let a developer with 1 month experience with Java publish any code. Because (most) newspapers and (at least some) software companies are finally out of the pioneer age while bloggers are still exploring their limits. Blogging seems to still be a slave to the “First Post�? phenomenon and many bloggers are swimming these unmarked waters, publishing hearsay, rumor, unsubstantiated opinion or just general under-researched posts. I predict that, eventually, as blogging matures and more options arise, blog readers will also learn to be more discriminatory and choose reputable bloggers that fact-check and care about quality not just agility. Quality is NOT a Bad Thing(tm), it’s merely a sign of maturity.

With community based tools and games spreading all over the net lately, I find myself thinking more and more about communities and ways to make them scale. Obviously thanks to the Internet we can create communities of scales never seen before. Physical limits no longer apply rather we are limited by human capacity to filter information and by the technology that helps us makes sense of this information.

I spent some time as a very active member of Consumating.com. During this time I learned to love the overheard feature. Overhears show on top of the screen and quote some random post from a random conversation. It’s a great way to discover conversations and people you didn’t know about so far but how does it scale? What happens when consumating grows to 100,000 users? A million? 650 million?

The simple solution is some static division of users: by geography, age, industry, etc. This solution seems to work pretty well in many cases especially if the division is appropriate to the context. Where this approach lacks, however, is when certain categories of the arbitrary division aren’t popular enough and users end up in what seems like an empty site or when some category becomes too popular and crowded. In such cases it might be better to use a dynamic, user-driven approach.

On tribe.net (and I’m told on Ning as well but I never bothered looking) any user can create a new group (or tribe) based on whatever topic (or lack thereof) she finds interesting. Some tribes are location based, some are interest based and some are community based. One of the pros of this system is that it is self regulating. Small tribes with little interest or activity tend to die off while tribes that get too big can fork off into several related tribes. Any user can start an offshoot of the parent tribe to concentrate on more specific topics or subsets of the community. Of course, this eventually brings us to a similar problem, what happens when you have too many tribes?

This problem becomes even worse on mobile. While on the web we can (somewhat) easily navigate lists or even hierarchical structures of groups, this is not so easy on a mobile phone. Navigating long lists is painful, and hierarchies will likely take too many page loads and end up in confusion. Perhaps a search based solution or a matching/recommendation engine would do better in this case. I’d love to hear about any ideas, experiments or even better, working solutions to this problem.

Note from the Avatar Psychology panel.

Speaking: Harvey Smith of Midway Studios Austin.

Topic is game avatar but extends to chat and other places where users create avatars.

You are the only one who knows yourself.
But you change with mood and time.
Different people have different ideas of you.
There’s probably someone else in the world that has your exact name
Etc.
So who are you?

A lot of the game experience goes on in the player’s mind.

Games let you express yourself either intentionally or through customization.
Players love self-expression.
Game avatars facilitate self-expression.

Goes through character generation through the ages starting with Mr. Potato head and paper dolls. Now you can do the same when buying cars online.

Participatory culture reflects a primal human drive. The future is in more and more customization.

RPG:

• Some players assume a different persona
• Some players express an idealized aspect of themselves.

FireTeam included player to player voice. Had existing characters. People would choose one that matches the type they like to play. When they added voice to the game suddenly the entire experience who the other people were changed because the voice did not match the avatar.

Avatars and Archetypes

• Avatars are masks
• Looking into your imago (your idealized expression)
• People tend to gravitate toward repeating patterns – archetypes.
• Archetypes provide for easily differentiated roles.
• Joseph Campbell – hero with a thousand faces
• Carl Jung - Multiple layers of identity
• The archetype IS the goal. Exmaples: Sam Fisher vs. Superman.

People intuitively get archetypes, they already think along those lines.

What do players identity choices mean?

• Avatar choice communicates a sense of personal identity or mood state.
• No objective meaning.
• Helps the player develop an understanding of self.
• Henry Jenkins quote: All of us move nomadically across the media landscape, cobbling together a personal mythology of symbols and stories.
• We appropriate symbols from the world and invest a personal meaning in them.
• Even in games without character creation
• Players make choices about what games to buy.
• Example: different boxes for WoW with different character types.

Identity is multi-layered

Double-consciousness during play.
What’s going on in the player’s mind while they’re playing?
the immersive fallacy – the player is the character but they’re also themselves. Leaving a blank slate for the player to project onto.

Game characters allow us to temporarily restructure our view of self.
Game experience can be intense and immersive but the avatar is still just a puppet. The player is aware of the avatar being a fictional construct.
Many aspects to identity during game play.
Player is aware of several things at once (achieving goals, own feelings, avatar’s feelings…)

Reinforcing identity in COD2 – repeating the stylized rituals that hint at your character identity repeatedly.

Identity Absorption

Scott McCloud (Understanding Comics): My car becomes an extension of my body, it absorbs my sense of identity, I become the car. “She hit me!�? not “her car hit my car�?

Identity is constructed
Judith Butler (Gender Trouble):

• Core aspects of identity are constructed.
• Coherence of gender categories seems natural
• Constructed via repeated stylized acts.
• Establishes the appearance of coherent gender.
• Gender/sexuality are performative.

Identity is constructed by repeated stylized acts.
Example: all chars in Diablo can heal themselves but they each doing it differently in a way that enhances their identity.

Masking in anime – protagonist are drawn in more vague lines, antagonist is in more detail. Can clearly see the more detailed version is not me.

Q: how important is emotion to creating avatar?
A: We’ll definitely see more and more of that as we have more computing resources available.

Q from Justin Hall: please talk some more about yourself (I’m paraphrasing here)
A: our current game is a shooter with a subversive political message.
Letting the player create a squad (so she connects to them more). Can create a new squad member when one dies. Initially designed with different characters having different world views.
On his own representation online: started as more emo and moved to more relaxed avatars.
Mentions a avatar that was created for him by users for a 2nd life interview.

Q: why do game designers insist on putting words in my mouth?
A: this is still an open question. Some designers still see this as a good thing.