Warrior Warrior Ancient and Medieval Rules
A Four Horsemen Enterprises Rules Set
  FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups AlbumAlbum   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Technical Issues: Fixing Yahoo Search

This forum is locked: you cannot post, reply to, or edit topics.   This topic is locked: you cannot edit posts or make replies.    Warrior Ancient and Medieval Rules Forum Index -> Egroup Archives
View previous topic :: View next topic  
Author Message
Mark Stone

Joined: 12 Apr 2006
Posts: 2076
Location: Buckley, WA

PostPosted: Mon Apr 10, 2006 2:53 am    Post subject: Technical Issues: Fixing Yahoo Search

In this message I offer a partial solution to the crappy condition of the Yahoo
Groups search function. It gets into some technical and policy detail about
website management, which most of you may not care about. But I do offer a step
by step description, at the end of this message, of how any of you can quickly
and easily search the archive of messages for this group from April 10, 2006
going forward. I'm still working on a solution for easily searching the archive
up to this date, but don't have a solution to offer yet.

Here's the reason it is so important to have at least this limited solution at
this time: once the new rule book is out, we effectively have a "blank slate"
when it comes to errata and clarifications. I think we'd all agree that (1) we
don't want Jon to have to answer new questions (and there will be new
questions) more than once, and (2) we want to be able to find Jon's answers to
new questions as easily as possible. While it would be ideal to have an easy
search for the whole group archive, it is most urgent to have easy search for
all of the archive starting with the publication of the new rule book. I can
offer at
least that limited solution.

Depending on your level of interest, you'll probably fall into one of three

(1) Don't care. "I don't care about all this technical stuff that has to do with
Yahoo Groups; I'm just here for Warrior." Great; then you don't need to read the
rest of this message.

(2) Don't care, but care that somebody cares. "I don't want the technical
background; I just want to be able to find stuff in the archive." That's fine;
you'll want to skip most of this message, and just read the step by step
instructions at the end.

(3) Care. "Why are we in this messed up situation with search, what has been
done, and what still needs to be done about it?" Read on.

So far, we've been caught between two problems: On the one hand, Yahoo has an
internal search of its own for Yahoo Groups, but it isn't very good. On the
other hand, Yahoo's privacy policies prevent external search engines like
Google from gaining access to archives in Yahoo Groups.

A bit more detail about each of these. The main problem with Yahoo's internal
search is that it doesn't search the entire archive of the group in one search.
You can only search limited date ranges at a time, meaning that if you really
want to search the entire archive you have to do dozens of searches manually.
I've actually done this a couple of times, and it sucks.

The privacy issue with external search engines (Google) is a bit different.
Partly there is some wisdom on Yahoo's part. This is part of their overall
strategy for keeping spam and spammers off of Yahoo Groups and away from its
participants. The privacy policy is part of how your email address is kept away
from would-be spammers, for example. Partly, however, this is just a business
tug of war between Yahoo and Google. Every time you use Yahoo's search for
something Yahoo makes money; every time you use Google's search for something
Google makes money. You can see why Yahoo might want to do its utmost to
prevent people from using Google Search to find Yahoo content.

One way to solve the problem would be to copy the Warrior Group archive over to
some location where Google Search could be pointed at it. This can't really be
done manually, since we now have an archive of 10s of thousands of messages. I
have written several "spider" scripts to try and automate this process (I
showed some preliminary results from this to Jon about a year ago), but not
surprisingly Yahoo has a number of ever-changing spider detection programs in
place to prevent just this kind of automated content scraping. So I've been
playing this cat and mouse game with Yahoo without much success for about a
year now ("spider", "cat", "mouse" -- yes, I know; I'm badly mixing my
metaphors at this point).

The further problem is that an external website archiving Yahoo Groups content
might be at best a temporary solution. I'm fairly sure that would be a
violation of Yahoo's terms of service, would be detected by Yahoo at some
point, and shut down via a "cease and desist" order.

What to do? Let me explain briefly what I've done, and what still remains to be
done, and then I'll explain in more detail what I've done so far.

One place that Google Search can legally get at Yahoo Groups content is via
Google's "Gmail" email service. Any individual is allowed to subscribe to a
Yahoo Group and have the Group emails sent to any email address. By sending
them to a Gmail address, you legally send them some place that Google Search
can get at them.

So essentially, what I've done is to create a fictitious individual (his name is
"Alexander Hannibal") who is a member of the Warrior Group, and subscribes to
this group via his Gmail account, which so happens to be "warriorplayers AT
gmail DOT com". If I share the login information for that account with this
group, then anyone in this group could log in and do a Google Search of all the
Warrior Group messages since Alexander Hannibal joined the Warrior Group (which
was today). And that's exactly what I'm going to do: share that login
information with the members of this group, provided no one abuses the
warriorplayers Gmail account.

What this doesn't help with is all those 10s of thousands of messages before
Alexander Hannibal joined our group. The correct solution would be to find a
way to copy those messages to Alexander Hannibal's Gmail account, which may be
skirting the borders of Yahoo's terms of service but is probably still legit. I
have found a program that is supposed to automate this process, but haven't had
time to see how well it works, or even if it works. It would need a fair degree
of customization in any case, and I certainly haven't had time for that. But I
hope to do all this at some point, and then we'll have complete Google Search
access to our own archive.

So: if you want to search any messages in the Warrior Group archive with a date
on or after April 10, 2006, here's how you do it step by step:

(1) Go to http://gmail.google.com/gmail
(2) When prompted to login, supply:
-- a username of "warriorplayers"
-- a p4ss w0rd of "Lancaster" (yes, upper and lower case matters)
(3) You should then be in the INBOX screen for "warriorplayers"
(4) At the top of this page you'll see a text entry box next to two buttons that
say, respectively, "Search Mail" and "Search the Web"
(5) Enter key words for what you're looking for in the archive into the text
entry blox, and then click on the "Search Mail" button
(6) Click through the search results until you find what you're looking for
(7) Click on the "Sign out" link when you're done.

It's my belief that all of us having access to this account will greatly
simplify life for Jon and others at FHE, and greatly reduce our frustration
with how hard it is to find stuff in our own archive. If, however, anyone from
FHE is uncomfortable with this approach, or if I find that this account is
being used for any purpose other than what I describe above, then I'll have it

Happy Searching,
Mark Stone

Back to top
View user's profile Send private message   MSN Messenger

Joined: 29 Mar 2006
Posts: 16447

PostPosted: Mon Apr 10, 2006 3:58 am    Post subject: Re: Technical Issues: Fixing Yahoo Search

This is a beauty of a post, Mark. But ultimately unnecessary. We'll be off
yahoo for good shortly

Stay tuned.....


[Non-text portions of this message have been removed]

Roll Up and Win!
Back to top
View user's profile Send private message  

Joined: 12 Apr 2006
Posts: 1412

PostPosted: Tue Apr 11, 2006 3:51 am    Post subject: Re: Technical Issues: Fixing Yahoo Search

So essentially, what I've done is to create a fictitious individual

Which might be a violation of the terms of service -- and certainly
will be if they learn about it.

I have simply saved every rules answer message from any of the authors.
I change the subject to begin with the word RULE or with the outline
number of the rule being discussed. Everything is in order in my
email Warrior mailbox folder.

Back to top
View user's profile Send private message  
Display posts from previous:   
This forum is locked: you cannot post, reply to, or edit topics.   This topic is locked: you cannot edit posts or make replies.    Warrior Ancient and Medieval Rules Forum Index -> Egroup Archives All times are GMT
Page 1 of 1

Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum

Powered by phpBB © 2001, 2002 phpBB Group