|
Roster Master for Dragonfly |
The Roster Master project has consistently offered functionality and innovative features well in advance of that of the EQ2Players site:
What is notable from this timeline is that EQ2Players introduces functionality that duplicates Roster Master well after such features are matured in the community project, while simultaneously disrupting the operation of the community project and removing features available to the community. The timing suggests that the Roster Master project was being used as a stalking horse for the SoE Web Services team, and that functionality was plagiarized from Roster Master and removed from the EQ2Players data feeds to prevent what what perceived as competition from the free open-source Roster Master project competing with the pay SoE Advanced Services features.
Bearing this in mind, on August 6, 2007, at the end of a series of e-mails prompted by a private message received from Carlos Mora, SoE Quality Assurance (Gninja), the following enhancement request and analysis of the then EQ2Players service was sent (using a lighthearted and informal tone). The intent was to request bugfixes and present a business case for replacing lost functionality and enhancing existing functionality of the service, specifically for the purpose of interoperability with Roster Master:
From: Dark_Grue
To: Gninja
Date: 08/06/2007 08:53:43
Subject: EQ2Players.com Roster Data Feed Enhancement RequestCarlos,
As I
threatenedpromised, here's some notes I typed up on the data feeds with some suggestions for enhancements/changes:XML Roster
First off, I'll say I'm not a big fan of the XML roster for two reasons: XML is too bulky to use for data transfer in my opinion, since the tags take up a lot of bandwidth (and if you indent the tags like the roster XML feed is, that just adds to the problem). The amount of overhead just in the tags can sometimes exceed the actual data content of an XML file! The other reason is that PHP doesn't include XML support by default. It has to be selected at the time that PHP is compiled, and since it requires external library support, it's not enabled by default. People using hosted services where PHP doesn't have XML support compiled in aren't uncommon. It's also a lot harder to write a XML parser, even with the XML API PHP provides (if the CSV roster was properly formatted, I can parse it just by exploding() each line into an array, or using a regexp - one line of code, instead of a whole handful of XML support and parsing functions).
I'm not necessarily advocating that the XML roster be removed, it's working now, and it doesn't do any harm, but my personal preference is the CSV roster, since it's the easiest to parse, and doesn't require special support.
CSV Roster
The CSV roster doesn't appear to conform to the CSV "spec", RFC 4180. First, the header line doesn't accurately represent the fields in the records. There is a "sex" column that does not exist in the data. The data also includes a lot of spurious whitespace (blank lines, tabs, and spaces), that are out of spec and do not belong. While it's not necessarily critical that the data file strictly follow RFC 4180 in terms of escaping characters (if the devs can confirm that no field can contain a comma, we can actually parse the data faster/easier without taking the full specification into account), the current format of the data is messy and wasteful. All that extra whitespace still counts as bytes of data sent each time the roster data is requested. That whitespace adds up as consumed bandwidth, which both ends of the connection are being billed for. It also means there's extra effort put into the parse stripping out blank lines and whitespace.
Roster Data
As things currently stand, the two data feeds and HTML rosters do not contain the same data, and do not contain the data that was available from EQ2Players.com prior to the EoF changes.
Guild id, name, and server: HTML and XML feed - wouldn't really work trying to shoehorn these into the CSV feed, and in the case of Guild id and name, those are configuration variables that Roster Master requires, so they're already available even before we query EQ2Players. Server's being parsed off of the HTML roster. So they're fine as-is.
name, prefixTitle , level, class, artisanClass, artisanLevel, guildStatus, dateJoined, guildRank, memberNumber - Both the XML and CSV rosters provide all these.
The CSV roster claims to have a "sex" (yes, please) column, but doesn't actually provide a field for it. It's in the header, but not in the data (there's not even a blank field for it). It'd be nice to actually have the data column added to both rosters.
characterId - This only appears in the HTML roster, and is missing from both the XML and CSV data feeds. This was one of the fields lost in the EoF changes, and it's absolutely vital in order to be able to create links to the individual character pages. We have to parse the HTML roster in order to reconstitute the links to the characters (but we'd have to do it anyway to get the guild summary information).
quests, KVD, lastonline, highestmeleehit, highestmagichit - Again, lost in the EoF changes. Last time Online and Quests Completed were both really good metrics for guild leaders to judge participation with, I'd really like to see them back.
Guild Summary Data
Right now, there are two things forcing us to pull the HTML page and parse it: getting the characterId, and the guild summary data (Guild Status, Guild Level, Date Formed, various statistics, etc.). If we could get just the statistics data as a second set of feeds (and again, I'd prefer not having to use XML for the reasons above), we could avoid that.
I'm not sure there's a convincing reason to create a new feed for that, as I don't know that the overhead of building the HTML page is that much different from parsing the data into a XML or CSV format (Roster Master just pulls the once page, and doesn't pull any of the CSS or graphical elements). We've got working parsers for the HTML page, so moving into a new feed would be creating new work for both of us. The critical problem is the severe performance problem with the HTML roster pages. However, if a new guild summary feed would alleviate our dependency on the HTML roster without us losing fields or requiring XML, I'm all for it.
Thanks!
-DG
No response from Mr. Mora was received after this message, despite that it was Mr. Mora that had initiated the contact, and requested the feedback. The message lays out the current problems that the Roster Master project was having with the EQ2Players data, and what we felt were ?roadmap? items for future enhancements. Those problems were exacerbated by the RoK release, not improved, and the suggestions put forth for returning data items or even expanding the data feeds were put straight into the paid Advanced Services.
The new pay guild roster service (part of the Advanced Guild Tools service which is $0.99/mo., or as part of a 5-service bundle for $5.99/mo.) and, based on the sorting column available in the EQ2Players roster demo, includes access to the following 34 data items on the roster. Compare this to the data items that are currently available to the community and data items that have been since removed from community access:
|
Advanced Guild Tools Service Data Items
|
SoE-provided XML, CSV Data Feeds
|
|
Character ID |
X |
|
Adventure Class |
X |
|
Adventure Level |
X |
|
Average Resists |
|
|
Guild Status |
Removed after EQ2Players RoK Release (3.4): November 12, 2007 |
|
Highest Heroic Opportunity Hit |
|
|
Highest Magical Hit |
Removed after Live Update #29: November 14, 2006 (EoF) |
|
Highest Melee Hit |
Removed after Live Update #29: November 14, 2006 (EoF) |
|
Items Crafted |
|
|
Kills vs Deaths Ratio |
Removed after Live Update #29: November 14, 2006 (EoF) |
|
Last Played |
Removed after Live Update #29: November 14, 2006 (EoF) |
|
Most Master Abilities |
|
|
Most NPC Kills |
|
|
Name |
X |
|
PvP Kills vs Death Ratio |
|
|
Quests Completed |
Removed after Live Update #29: November 14, 2006 (EoF) |
|
Race |
|
|
Rank |
X |
|
Rares Harvested |
|
|
Secondary Tradeskill Level |
X |
|
Secondary Tradeskill Prof |
X |
|
PvP Death Streak |
|
|
PvP Total Deaths |
|
|
PvP Kill Streak |
|
|
PvP Total Kills |
|
|
Total Deaths |
|
|
Total Health |
|
|
Total Item Discoveries |
|
|
Total Items Discovered on Server |
|
|
Total Power |
|
|
Total Status |
|
|
Tradeskill Level |
X |
|
Tradeskill Prof |
X |
In addition, the Date Joined and Prefix Title fields were removed after EQ2Players RoK Release (3.4) on November 12, 2007. Between the EoF and RoK releases (a span of a year), a total of eight different data items were removed from the SoE-provided data feeds provided to the customer community. In both cases, the changes to the data feed directly impacted both the development of the Roster Master project (by requiring significant redevelopment efforts in order to adapt to the changes and work around the problems with the EQ2Players data), as well as the significant reduction in quality and amount of data that was available to the project for processing.