What if the internets gets shut down forever?

Started by shmax, March 10, 2009, 09:11:18 AM

previous topic - next topic
Go Down

shmax

March 10, 2009, 09:11:18 AM Last Edit: December 31, 1969, 04:00:00 PM by Guest
[wrap=left:2o2nt3dd][/wrap:2o2nt3dd]I get it. You guys don't trust me to keep the site around for as long as your collection is. I mean who knows, I could be hit by a bus the next time I go outside, and that only gives us a few months to play around with. But fear not, as always I'm here to assuage your silly little fears. You'll now notice a "Download Collection" link in the Membership box, just like you've both been clamoring for. As always, please drop by the forums with any suggestions or problems you come across.

Encore_Prime

#1
March 11, 2009, 02:55:14 AM Last Edit: December 31, 1969, 04:00:00 PM by Guest
very cool, thank you.
<a href="http://www.shmax.com/img/sigs/signature.swf?user_id=483" target="_blank" class="new_win">http://www.shmax.com/img/sigs/signature.swf?user_id=483</a>

Waverider

#2
March 21, 2009, 01:31:01 PM Last Edit: December 31, 1969, 04:00:00 PM by Guest
Thank you. This comes in very handy.  :D
<a href="http://www.shmax.com/img/sigs/signature.swf?user_id=478" target="_blank" class="new_win">http://www.shmax.com/img/sigs/signature.swf?user_id=478</a>

Chaotic Descent

#3
June 05, 2009, 09:42:52 PM Last Edit: December 31, 1969, 04:00:00 PM by Guest
Could we have the ability to download the entire database rather than only the list of figures we personally own? That way we could theoretically update the downloadable version, or use it for other things. I currently had an idea for an online database of TF repaints and mods that would save a lot of time if existing TFs were already listed. It also would allow us to make easy lists of... for example, figures we don't have but want.

shmax

#4
June 05, 2009, 09:55:01 PM Last Edit: December 31, 1969, 04:00:00 PM by Guest
Quote from: "Chaotic Descent"
Could we have the ability to download the entire database rather than only the list of figures we personally own? That way we could theoretically update the downloadable version, or use it for other things. I currently had an idea for an online database of TF repaints and mods that would save a lot of time if existing TFs were already listed. It also would allow us to make easy lists of... for example, figures we don't have but want.
I'll have to ponder that one. It's taken years to assemble all this information, and I'm not sure it would really be fair to all the people who did the work to simply start providing users with complete dumps of the data set so they can go off and start their own websites! What did you have in mind, were you going to write a program to parse xml/csv, and try to import all the info? If you're only interested in repaints, how would you extract only the repaints from the data set?

So, I dunno. Maybe if you explain to me exactly what it is you want to do with the data we can work something out in private, but for now I can't really see the appeal in doing what you suggest.

Chaotic Descent

#5
June 06, 2009, 12:47:37 AM Last Edit: December 31, 1969, 04:00:00 PM by Guest
Quote from: "shmax"
Quote from: "Chaotic Descent"
Could we have the ability to download the entire database rather than only the list of figures we personally own? That way we could theoretically update the downloadable version, or use it for other things. I currently had an idea for an online database of TF repaints and mods that would save a lot of time if existing TFs were already listed. It also would allow us to make easy lists of... for example, figures we don't have but want.
I'll have to ponder that one. It's taken years to assemble all this information, and I'm not sure it would really be fair to all the people who did the work to simply start providing users with complete dumps of the data set so they can go off and start their own websites! What did you have in mind, were you going to write a program to parse xml/csv, and try to import all the info? If you're only interested in repaints, how would you extract only the repaints from the data set?
*sigh* Well that's surprising and disappointing that I've found yet another project that considers a list of TFs to be data that's too valuable to be public. EVERY SINGLE DAMNED TF ARCHIVING SITE is like this! Every single damned one of them has to start over from scratch every single time a new one pops up. because it was so much work, no one wants to share it. no, you have to do the same work over again. TFU.info, TF wiki, shmax, and more before that. Not to mention the numerous people who have made collecting guide books. (there has to be at least 3 different series of those)
Your downloaded lists are already stripped of most of the extraneous fields, and yet you're still hording a complete list? Name, series, faction, year (, sizeclass, other variant distinguishing label such as store exclusivity). Making that public would not be fair to the people who did data entry? Do they have a say in the matter?

Quote
So, I dunno. Maybe if you explain to me exactly what it is you want to do with the data we can work something out in private, but for now I can't really see the appeal in doing what you suggest.
Oops. I meant FAN repaints and mods.

But it's moot since I have other priorities currently that would prevent me from doing such a project. (a diplomatic way of saying I'm struggling just to take care of myself, which is why I've never submitted any photos here.)

shmax

#6
June 06, 2009, 06:37:23 AM Last Edit: December 31, 1969, 04:00:00 PM by Guest
Quote from: "Chaotic Descent"
*sigh* Well that's surprising and disappointing that I've found yet another project that considers a list of TFs to be data that's too valuable to be public.
Easy, partner. I really didn't have a choice but to start from scratch--before shmax.com there weren't any other Transformer databases of comparable size and complexity. For what it's worth this IS a collaborative project--I write all the code, but for the past year all of the serious data entry has come from the site's users. Everyone is welcome to contribute records or modify existing information--if there's some obscure toyline you specialize in, you don't have to start your own website from scratch to do it justice, you can submit your info and photos to the Shmax Database and sort of claim that area as your own. The information stored here IS public--nothing's hidden from view, and everything's free (and there are no ads). But if I put a big button marked "Download the Shmax Database!" somewhere, well, what would be the point in a general sense? This is a website, and it's designed to be viewed a page at a time. If you tried to print it out (SKUs and parts) you'd have around what, 500 pages? The only conceivable purpose anyone could have for the entire raw data set would be to try to import it into their own database, and that's just too small of an audience to justify the feature (there would be technical issues to overcome--file size limits, memory issues, bandwidth, etc). I'll also mention that anyone prepared to import that much data in one fell swoop will have already done a lot of engineering, and presumably have the skill to simply data mine the information automatically. I won't make any special effort to stop you if that's what you want to do. In fact, the code you write to process the data--whether it comes from data mining or an XML or Excel data dump--would be much the same. You could also just cheat and simply add every single toy to your collection, and then download that.

One thing I can tell you if you're serious about all this is that if you haven't started development yet, it's way too early to be worrying about where the data is going to come from--that will be the least of your worries compared to designing your schema, developing a user interface, figuring out how to handle all the variants and different releases from different countries, preparing a robust development pipeline, etc. Or maybe you have something very simple in mind, I don't know. If you really just want a simple list of the name of every known Transformer toy, well sure, I can provide that for ya. Maybe we can barter for it--you know how I am about photos!

So don't be angry, just go ahead and do what you want to do. I actually think that a database of fan-customized toys is a great idea, and in fact it's something we've been talking about doing here. Before talking to Hellscream I never realized how popular the hobby is, but I'm starting to get the picture!

Chaotic Descent

#7
June 07, 2009, 09:09:12 PM Last Edit: December 31, 1969, 04:00:00 PM by Guest
TL;DR: Yeah. I'm probably just being selfish and lazy and letting all failures of progress to discourage me.

Quote from: "shmax"
Everyone is welcome to contribute records or modify existing information
*snip*
The information stored here IS public--nothing's hidden from view, and everything's free (and there are no ads). But if I put a big button marked "Download the Shmax Database!" somewhere, well, what would be the point in a general sense? This is a website, and it's designed to be viewed a page at a time. If you tried to print it out (SKUs and parts) you'd have around what, 500 pages? The only conceivable purpose anyone could have for the entire raw data set would be to try to import it into their own database, and that's just too small of an audience to justify the feature (there would be technical issues to overcome--file size limits, memory issues, bandwidth, etc).
No, not the entire raw data. I already said the format of your download option strips out plenty of fields, which is fine, but only listing the figures you already have is a huge limitation. You don't need to be able to do fancy database programming to sort a single spreadsheet table file either.
When your collection gets so big that you can't keep track of it, you don't want to know what figures you DO have, you want to know what figures you DON'T have.
I went to BotCon this year and accidentally bought a 2nd boxed Max-B figure. I had to look up which of the Japanese BW flipchanger jets I already owned so that I could tell which one I still needed to get. I STILL don't know if I have Micron Booster wave 1, and I saw it at BotCon but passed because it wasn't listed as something I needed in one of my forum threads. It's possible it was mistakenly omitted. I still can't remember which Micromasters I'm missing because I'm near the end of my collection.

Quote
I'll also mention that anyone prepared to import that much data in one fell swoop will have already done a lot of engineering, and presumably have the skill to simply data mine the information automatically. I won't make any special effort to stop you if that's what you want to do. In fact, the code you write to process the data--whether it comes from data mining or an XML or Excel data dump--would be much the same. You could also just cheat and simply add every single toy to your collection, and then download that.

One thing I can tell you if you're serious about all this is that if you haven't started development yet, it's way too early to be worrying about where the data is going to come from--that will be the least of your worries compared to designing your schema, developing a user interface, figuring out how to handle all the variants and different releases from different countries, preparing a robust development pipeline, etc. Or maybe you have something very simple in mind, I don't know. If you really just want a simple list of the name of every known Transformer toy, well sure, I can provide that for ya. Maybe we can barter for it--you know how I am about photos!
You already know that's all outside my abilities. and adding every single toy to my collection would take a long time just to get those results. Maybe not as much time as doing the data entry yourself manually. Maybe not as much time as it would take to learn how to "data mine" from this site.

Ehh... aren't there systems where you can just "plug in" a table file(s)? I dunno.

That doesn't even cover uses like printing out a list to bring with you to BotCon so you can keep track of what you're looking for. For example, what if you just want a list, instead of pictures (especially if most of the pictures are missing) Maybe you want to fit more than 8 figures per page you print out. Huh. ... the "details" display mode is actually more compact than the "summary" mode. Go figure.

I also want to start keeping track of my collection's multiples, and not on separate accounts, for the purpose of my hypothetical dream museum. For that I need to keep track of 1 set for my personal collection, both wanted and owned, and at least one if not two sets for the museum. (ok, by the time I got as far as collecting 1 of everything + 3rd+ modes + my small personal collection, I'd probably be doing serious work on this and be asking for support and probably have plenty of photographs to submit)

Quote
So don't be angry, just go ahead and do what you want to do. I actually think that a database of fan-customized toys is a great idea, and in fact it's something we've been talking about doing here. Before talking to Hellscream I never realized how popular the hobby is, but I'm starting to get the picture!
I don't even know how... All I know is that I can't keep track of the pictures I have. There's always a fan repaint or mod I'm thinking of and I can't find the damned picture to see exactly what it looks like.
The mold/name/alt-mode relationships between original toys also helps with fan repaints/mods, especially when you're trying to find a TF to use in a repaint/mod.


I just... I can't do very much. and I wanted to start keeping track of my multiples. but I can't. so it's discouraging.
There's really no point in asking YOU to implement a feature that only I want to use. and... I don't know. Maybe I'm being selfish in wanting something for nothing, but it feels like twice as much work. I gotta do work before I can start work on my own project.
TL;DR: Yeah. I'm probably just being selfish and lazy and letting all failures of progress to discourage me.

shmax

#8
June 07, 2009, 11:09:21 PM Last Edit: December 31, 1969, 04:00:00 PM by Guest
Quote from: "Chaotic Descent"
No, not the entire raw data. I already said the format of your download option strips out plenty of fields, which is fine, but only listing the figures you already have is a huge limitation. You don't need to be able to do fancy database programming to sort a single spreadsheet table file either.
When your collection gets so big that you can't keep track of it, you don't want to know what figures you DO have, you want to know what figures you DON'T have.
I went to BotCon this year and accidentally bought a 2nd boxed Max-B figure. I had to look up which of the Japanese BW flipchanger jets I already owned so that I could tell which one I still needed to get. I STILL don't know if I have Micron Booster wave 1, and I saw it at BotCon but passed because it wasn't listed as something I needed in one of my forum threads. It's possible it was mistakenly omitted. I still can't remember which Micromasters I'm missing because I'm near the end of my collection.
Hmm, I guess I misunderstood, from your last posts I somehow got the impression that you wanted the data so you could launch a website. Asking for the ability to download your "Missing" list is a little more reasonable, especially if it's just a simple list, like you say. But I don't understand how this would have helped you with the "Max-B" figure situation. Why didn't you just print out your collection list and bring it along with you? You could have flipped to the Beast Wars section, seen that you already own him, and moved on to the next table.

That said, the collection functionality is all still new, and if I'm missing features it's only because nobody ever asks for them! I do take requests when they're reasonable and well thought-out, but well, it's damn rare that anybody ever provides any feedback.

Quote
You already know that's all outside my abilities.
It was outside of mine, too, a few years ago. A good way to learn fast is to just dive right in on a project you're passionate about, and brother, you've got plenty of passion.

Quote
Ehh... aren't there systems where you can just "plug in" a table file(s)? I dunno.
Not sure I understand what you mean. What's a "table file"?

Quote
That doesn't even cover uses like printing out a list to bring with you to BotCon so you can keep track of what you're looking for.
Hmm, good point. Maybe I should see if I can come up with a way to print the wish list.

Quote
For example, what if you just want a list, instead of pictures (especially if most of the pictures are missing) Maybe you want to fit more than 8 figures per page you print out. Huh. ... the "details" display mode is actually more compact than the "summary" mode. Go figure.
Yeah, I can see where you're coming from. I'll give it some thought, okay? And yes, "details" mode is more compact than "summary" mode. Maybe I should call one "expanded" and the other "table". I dunno.

Quote
I also want to start keeping track of my collection's multiples, and not on separate accounts, for the purpose of my hypothetical dream museum. For that I need to keep track of 1 set for my personal collection, both wanted and owned, and at least one if not two sets for the museum. (ok, by the time I got as far as collecting 1 of everything + 3rd+ modes + my small personal collection, I'd probably be doing serious work on this and be asking for support and probably have plenty of photographs to submit)
Well, of course with the new system you can already add multiples. Why not just add everything to one account, and simply note in the comments which ones are intended for the "museum"?

Quote
I don't even know how... All I know is that I can't keep track of the pictures I have. There's always a fan repaint or mod I'm thinking of and I can't find the damned picture to see exactly what it looks like.
The mold/name/alt-mode relationships between original toys also helps with fan repaints/mods, especially when you're trying to find a TF to use in a repaint/mod.
Well, if you don't mind waiting a bit we will eventually get around to letting users supply their own photos for items in their collection. And once we get that far, it will be fairly trivial to add support for tagging items as "customized". So, we'll get there, just gotta be patient.

Chaotic Descent

#9
June 11, 2009, 11:47:03 AM Last Edit: June 16, 2009, 10:37:30 AM by Chaotic Descent
Quote from: "shmax"
Hmm, I guess I misunderstood, from your last posts I somehow got the impression that you wanted the data so you could launch a website.
I had more than one idea I presented as examples of the use of being able to download the database.

Quote
Asking for the ability to download your "Missing" list is a little more reasonable, especially if it's just a simple list, like you say. But I don't understand how this would have helped you with the "Max-B" figure situation. Why didn't you just print out your collection list and bring it along with you? You could have flipped to the Beast Wars section, seen that you already own him, and moved on to the next table.
Because, as I said, it's less efficient. If you have more figures than you're looking for, it's a larger list. While you can use the process of elimination, it may take longer to look through your list to find the right section and the right figure, and you'll have to look through the whole list of figures in the series to make sure it's not there, rather than just locate the one figure you see the dealer has to see if it's one you need.

Quote
That said, the collection functionality is all still new, and if I'm missing features it's only because nobody ever asks for them! I do take requests when they're reasonable and well thought-out, but well, it's damn rare that anybody ever provides any feedback.
Sorry about that. I tend to be a non-person most of the time, unable to contribute anything to anyone.

Quote
Quote
You already know that's all outside my abilities.
It was outside of mine, too, a few years ago. A good way to learn fast is to just dive right in on a project you're passionate about, and brother, you've got plenty of passion.
It's possible I do, although I seem to be easily discouraged. I haven't touched databases in months.

Quote
Quote
Ehh... aren't there systems where you can just "plug in" a table file(s)? I dunno.
Not sure I understand what you mean. What's a "table file"?
... uh... XML? isn't that a table file? table/spreadsheet? jargon?

Quote
Quote
That doesn't even cover uses like printing out a list to bring with you to BotCon so you can keep track of what you're looking for.
Hmm, good point. Maybe I should see if I can come up with a way to print the wish list.
Or to be able to print out owned, and wish list, and both together. (and I'd suggest if both are together, to have the items interspersed and sorted the way they normally would be if they weren't sorted by have/want, otherwise there's no advantage to printing them together other than saving paper by starting one as soon as the other finishes. although you could also have all the owned figures within a series/year/sizeclass listed first before the wish list figures were listed. That's plenty of options.

Quote
Quote
For example, what if you just want a list, instead of pictures (especially if most of the pictures are missing) Maybe you want to fit more than 8 figures per page you print out. Huh. ... the "details" display mode is actually more compact than the "summary" mode. Go figure.
Yeah, I can see where you're coming from. I'll give it some thought, okay? And yes, "details" mode is more compact than "summary" mode. Maybe I should call one "expanded" and the other "table". I dunno.

Quote
I also want to start keeping track of my collection's multiples, and not on separate accounts, for the purpose of my hypothetical dream museum. For that I need to keep track of 1 set for my personal collection, both wanted and owned, and at least one if not two sets for the museum. (ok, by the time I got as far as collecting 1 of everything + 3rd+ modes + my small personal collection, I'd probably be doing serious work on this and be asking for support and probably have plenty of photographs to submit)
Well, of course with the new system you can already add multiples. Why not just add everything to one account, and simply note in the comments which ones are intended for the "museum"?
Ah. Thanks. I realized this after I posted that. Although I would still like to download the list so that I can see just how complete my collections are, since the total for each is different. My museum collection is only working towards getting figures with 3rd modes, while my personal collection is just stuff I absolutely have to have.

Quote
Quote
I don't even know how... All I know is that I can't keep track of the pictures I have. There's always a fan repaint or mod I'm thinking of and I can't find the damned picture to see exactly what it looks like.
The mold/name/alt-mode relationships between original toys also helps with fan repaints/mods, especially when you're trying to find a TF to use in a repaint/mod.
Well, if you don't mind waiting a bit we will eventually get around to letting users supply their own photos for items in their collection. And once we get that far, it will be fairly trivial to add support for tagging items as "customized". So, we'll get there, just gotta be patient.
Wouldn't it be best to host the majority of a database like that on another server, or can you really handle such a large amount of data? I mean, the possibilities are enormous. You could add bootlegs, which would probably double or triple the database. Fan customs. What if you added corresponding fiction entries and character pages and continuities? (making it more like TFwiki, I guess) and you already mentioned other toy lines, not even necessarily Transformers related. We could do all the Takara TF-like lines, like the entirety of Brave, Machine Robo Rescue & Mugenbine, and the wee lines of Web Diver & DaiGander(sp?). Go-Bots, like TFU.info has.

Do you want to host everything, or do you think perhaps if it got too huge you would consider linking off-site for different content? I guess it's too early to really make any decision like that.

I think what I really need is some kind of offline image database. That way it would be very fast to react, but I'd still be able to see images instead of just lists.
Unfortunately, I don't think image databases are standardized and as user-friendly as the very OS you use. I think I looked into this before on wikipedia. :S (for my porn collection. :) I think it might be just a little bit bigger than my TF image collection. :> )

shmax

#10
June 13, 2009, 11:27:35 AM Last Edit: December 31, 1969, 04:00:00 PM by Guest
hey CD, you raise a lot of good points. I'm still in the middle of another project that's eating up my time, but it's almost done, and when I finish I'll come back and try to respond to you in more depth. For now, I can tell you that I think we'll be able to come up with something to make you happy (or as close as you ever get!), so hang in there.

shmax

Go Up