bug#24052: Bing : crawling issues with www.gnu.org

bug#24052: Bing : crawling issues with www.gnu.org

[Fabrice Canel]

[-- Attachment #1: Type: text/plain, Size: 894 bytes --]

Contacting now bug-gnu-emacs@gnu.org<mailto:bug-gnu-emacs@gnu.org> as I didn't receive a reply from webmasters@gnu.org<mailto:webmasters@gnu.org>.
Who can help indexing your content and other pages on gnu.org?

Thanks,
Fabrice

From: Fabrice Canel
Sent: Tuesday, July 19, 2016 12:49 AM
To: 'webmasters@gnu.org' <webmasters@gnu.org>
Subject: Bing : crawling issues with www.gnu.org

Good day,

We (Bing) have an major issue crawling and indexing your site.

Our customers alerted us that https://www.gnu.org/software/emacs/ cannot be found in our results.
Investigating we believe that you are preventing bingbot crawling from our main IP ranges, we don't have issue fetching outside our IP ranges.

Can you investigate? I am willing to share our crawler bingbot IP ranges if this can help fixing this issue.
Thanks,
Fabrice Canel
Principal Program Manager
Microsoft Bing

[-- Attachment #2: Type: text/html, Size: 3616 bytes --]

bug#24052: Bing : crawling issues with www.gnu.org

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 521 bytes --]

Fabrice Canel <Fabrice.Canel@microsoft.com> writes:

> We (Bing) have an major issue crawling and indexing your site.

Hi Fabrice,

> Our customers alerted us that https://www.gnu.org/software/emacs/
> cannot be found in our results.  Investigating we believe that you are
> preventing bingbot crawling from our main IP ranges, we don't have
> issue fetching outside our IP ranges.

Do you know if the issue only happens with the software/emacs/ pages, or
does it affect all pages from https://www.gnu.org?

Cheers,
Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

bug#24052: Bing : crawling issues with www.gnu.org

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 512 bytes --]

Fabrice Canel <Fabrice.Canel@microsoft.com> writes:

Hi again Fabrice,

I confirm the issue, all pages under software/emacs/ seem to be missing
from search results.

Also, I could not find the index page of https://www.gnu.org by
searching for "GNU" or "GNU Operating System", however, searching for
"GNU Guile", I could find in the search results the Guile homepage at
https://www.gnu.org/software/guile/guile.html, but it was still pretty
far down in the first search result page.

Cheers,
Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

bug#24052: Bing : crawling issues with www.gnu.org

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 265 bytes --]

Hi Fabrice,

Does this also explain why the Emacs Lisp manual isn't indexed by Bing?

Searching for "emacs lisp other display specs", for example, does not return https://www.gnu.org/software/emacs/manual/html_node/elisp/Other-Display-Specs.html

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24052: Bing : crawling issues with www.gnu.org

[Fabrice Canel]

Hi Clément, Nicolas,

Yes, we have major issue indexing this site. This is not limited to this URL.
Any contact in webmasters@gnu.org?

Thanks,
Fabrice

-----Original Message-----
From: Clément Pit--Claudel [mailto:clement.pit@gmail.com] 
Sent: Friday, July 22, 2016 1:06 PM
To: bug-gnu-emacs@gnu.org; Fabrice Canel <Fabrice.Canel@microsoft.com>
Subject: Re: bug#24052: Bing : crawling issues with www.gnu.org

Hi Fabrice,

Does this also explain why the Emacs Lisp manual isn't indexed by Bing?

Searching for "emacs lisp other display specs", for example, does not return https://www.gnu.org/software/emacs/manual/html_node/elisp/Other-Display-Specs.html

Clément.

bug#24052: Bing : crawling issues with www.gnu.org

[Ruben Rodriguez]

I've checked the logs and I see access from bingbot to all our sites at
the machine where gnu.org and nongnu.org lives. I also checked the
robots.txt and they look ok.

Could you provide more details on failure to fech from bingbot, so I can
look in the logs? If so, please send it to sysadmin at gnu.org

Cheers,
-- 
Ruben Rodriguez | Senior Systems Administrator, Free Software Foundation
GPG Key: 472F4409 | https://fsf.org | https://gnu.org

bug#24052: Bing : crawling issues with www.gnu.org

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 359 bytes --]

Ruben Rodriguez <ruben@fsf.org> writes:

> Could you provide more details on failure to fech from bingbot, so I can
> look in the logs? If so, please send it to sysadmin at gnu.org

Hi Ruben,

I don't think this is specific to Emacs, it seems to affect all webpages
at www.gnu.org.  Is it ok if I close this ticket and let you handle the
issue?

Cheers,
Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

bug#24052: Bing : crawling issues with www.gnu.org

[Ruben Rodriguez]

On 07/28/2016 03:22 PM, Nicolas Petton wrote:
> Ruben Rodriguez <ruben@fsf.org> writes:
> 
>> Could you provide more details on failure to fech from bingbot, so I can
>> look in the logs? If so, please send it to sysadmin at gnu.org
> 
> Hi Ruben,
> 
> I don't think this is specific to Emacs, it seems to affect all webpages
> at www.gnu.org.  Is it ok if I close this ticket and let you handle the
> issue?

Yes, that's fine.

-- 
Ruben Rodriguez | Senior Systems Administrator, Free Software Foundation
GPG Key: 472F4409 | https://fsf.org | https://gnu.org

bug#24052: Bing : crawling issues with www.gnu.org

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 271 bytes --]

Ruben Rodriguez <ruben@fsf.org> writes:

>> I don't think this is specific to Emacs, it seems to affect all webpages
>> at www.gnu.org.  Is it ok if I close this ticket and let you handle the
>> issue?
>
> Yes, that's fine.

Thanks, closing the issue then.

Cheers,
Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]