MyBB Community Forums

This is the OP from the thread references in the first post. Been away for a while, but recently came back to give this another try. This time, to rule out Mac-specific issues, I'm doing the merge on an Ubuntu 10.04 box. I am running MyBB 1.6 and merge 1.6.0. And I continue to have the very same issues as before (see here: http://community.mybb.com/thread-68812.html )

This time, however, the progress pages provide a little more insight, perhaps. The problem seems to be with mime_content_type(), as I get the following error for each attachment:

Warning [2] mime_content_type(): File or path not found 'jpg_thumb' -
Line: 123 - File: convert/boards/smf/attachments.php PHP 5.3.2-1ubuntu4.5 (Linux)

File	Line	Function
[PHP]	 	errorHandler->error
/convert/boards/smf/attachments.php	123	mime_content_type
/convert/resources/modules/attachments.php	39	SMF_Converter_Module_Attachments->convert_data
/convert/boards/smf/attachments.php	98	Converter_Module_Attachments->insert
/convert/index.php	970	SMF_Converter_Module_Attachments->import

Where "jpg_thumb" in the first varies with the filename extension in the SMF attachments table (i.e., it will take the value "jpg", "png", "jpg_thumb", etc). Just a guess, but it looks like the attachements merge function is munging the filename somehow?

Indeed, looking at the file, you are only passing the file extension to mime_content_type, and mime_content_type requries the full filename with path! Looks an easy fix…

---

Quick update: Replaced line 123 of boards/smf/attachments.php with

$insert_data['filetype'] = mime_content_type($import_session['uploadspath'].'/'.$data['filename']);

And still got errors: At this point in the execution, $import_session['uploadspath'] is empty!

Warning [2] mime_content_type(): File or path not found '/bus11-3.jpg' - Line: 123 - File: convert/boards/smf/attachments.php PHP 5.3.2-1ubuntu4.5 (Linux)
File	Line	Function
[PHP]	 	errorHandler->error
/convert/boards/smf/attachments.php	123	mime_content_type
/convert/resources/modules/attachments.php	39	SMF_Converter_Module_Attachments->convert_data
/convert/boards/smf/attachments.php	98	Converter_Module_Attachments->insert
/convert/index.php	970	SMF_Converter_Module_Attachments->import

Unsure how best to proceed at this point.

(2010-09-22, 05:32 PM)CaptOblivious Wrote: [ -> ]And still got errors: At this point in the execution, $import_session['uploadspath'] is empty!

Warning [2] mime_content_type(): File or path not found '/bus11-3.jpg' - Line: 123 - File: convert/boards/smf/attachments.php PHP 5.3.2-1ubuntu4.5 (Linux)
File	Line	Function
[PHP]	 	errorHandler->error
/convert/boards/smf/attachments.php	123	mime_content_type
/convert/resources/modules/attachments.php	39	SMF_Converter_Module_Attachments->convert_data
/convert/boards/smf/attachments.php	98	Converter_Module_Attachments->insert
/convert/index.php	970	SMF_Converter_Module_Attachments->import

Unsure how best to proceed at this point.

i fixed that by putting

global $import_session;

into the function, although it doesnt help the slightest, because your refering then to a file which does not exist, as "filename" is the original filename of the attachment, but smf puts them down in the directory in some naming scheme, so your refering to the wrong "filename" up there, guess there is a different one from the database pulled which would be the correct one.

The error I get now lets me question the whole "put the domain path" in it, as "uploadpath" would then be http://.... and I doubt the php function can work with that? Guess that only works with filenames and directories from the filesystem, but then I have no clue about php whatsoever, just guessing here in pure desperation :-)

(2010-09-23, 01:45 PM)Tupsi Wrote: [ -> ]it doesnt help the slightest, because your refering then to a file which does not exist, as "filename" is the original filename of the attachment, but smf puts them down in the directory in some naming scheme, so your refering to the wrong "filename" up there, guess there is a different one from the database pulled which would be the correct one.

That's a good point. I didn't stop to think why the merge would be checking the MIME type of a non-existant file. So it looks like the problem runs even deeper than I'd suspected.

I'm beginning to think that the entirety of smf/attachments.php is quite a mess…

---

Additional findings:

changing line 123 of attachments.php to

$insert_data['filetype'] = mime_content_type($import_session['uploadspath'].$this->generate_raw_filename($data));

and declaring $import_session as global at the beginning of the function (thanks!!) gets the paths correct, and mime_content_type has stopped complaining…except in two cases.

In the first, the function convert_data() is, for some reason, iterating over avatars as well as genuine attachments, and generate_raw_filename() naturally doesn't produce the correct filename for avatars.

In the second, there is a problem in that SMF appears to have changed the way it generates attachment filenames at some point.

My installation of SMF uses two systems for generating attachment filenames. The first is a munging together of the attachment ID, the filename, and a generated hash; generate_raw_filenames() correctly produces the filename in these cases. However, in some cases, the filename is simply the id and a hash concatenated together with an underscore: "ID_hash". Generate_raw_filenames() get this case wrong; the hashing method is different, too, but the good news is that in these cases, the hash can be retrieved from the database.

Moreover, for those attachements that mime_content_type has ceased complaining about still do not copy over the the MyBB installation, with the usual errors, although I think I might have an idea why now. Will post a follow-up (and some bug reports) shortly.

---

Since I'm using an absolute path to locate my SMF attachments, this bug report was useful in figuring out what was going wrong:
http://dev.mybb.com/issues/1155

Applying the resulting changes to merge_fetch_remove_file in convert/functions.php (which were not included in Merge v1.6.0) fixed the problems for the files whose filenames are being correctly generated.

What remains is to patch generate_raw_filenames to get the rest right, and to filter out avatars from the attachments merge process.

---

And of course, the new hashing scheme was taken care of at:
http://dev.mybb.com/issues/1156

I'm really not on top of things today, am I? Need to spend more time at the bug tracking website.

Sorry for the multitude of largely useless updates. It looks like the only real outstanding bug that I can see and that hasn't been addressed yet is the problem with the parameters to mime_content_type(); the fix above is working fine for me.

---

Alright, I've finally gotten my attachments to copy over correctly. I'm using the trunk version of the merge, so as to get all of the various bug fixes for attachments. I've made the following changes to boards/smf/attachments.php:

in function convert_data()
add:

global $import_session;

change

$insert_data['filetype'] = mime_content_type(get_extension($data['filename']));

to

$insert_data['filetype'] = mime_content_type($import_session['uploadspath'].$this->generate_raw_filename($data));

in generate_raw_filename()
change

if(isset($attachment['file_hash']))

to

if(isset($attachment['file_hash']) && $attachment['file_hash']!='')

(in installations like mine, there are a mix of old-style and new-style attachment entries, so simply checking for the file_hash field isn't sufficient: You have to be sure it isn't empty, either.)

also change the two lines

return $attachment['ID_ATTACH']."_".$attachment['file_hash'];
…
return $attachment['ID_ATTACH']."_".str_replace('.', '_', $attachment['filename']).md5($attachment['filename']);

to

return "/".$attachment['ID_ATTACH']."_".$attachment['file_hash'];
…
return "/".$attachment['ID_ATTACH']."_".str_replace('.', '_', $attachment['filename']).md5($attachment['filename']);

This last change may be a dirty hack; when using an absolute path or relative path (rather than a URL, which I have not tested), $import_session['uploadspath'] does not contain a trailing slash. A better fix would probably be to check whether it does, and then add one if it is missing. Anyway, without the trailing slash, every instance of:

$import_session['uploadspath'].$this->generate_raw_filename($data)

will generate incorrect results.

---

Final note for today: Although my attachments are largely copying over correctly, there are two outstanding issues:

1) some files were corrupted. Ugh!

2) the post id's assigned to each attachment are very, very off. Although the post IDs change when posts are imported from SMF to MyBB, the post ID of each attachment remains the same. Thus, in my SMF installation, I have a post ID 33833 with two attachments. In my merged MyBB installation, the same post is now ID 33097, but the associated attachments are still attached to MyBB post 33833 (which is, as you might expect, on a completely different topic).

is it possible to get read access to the svn in order to checkout the files? Clicking every single file in redmine seems so silly...

Capt: do you get Avatars now copied over? So far I never got any single one in my attempts.

(2010-09-27, 07:49 AM)Tupsi Wrote: [ -> ]is it possible to get read access to the svn in order to checkout the files? Clicking every single file in redmine seems so silly...

Capt: do you get Avatars now copied over? So far I never got any single one in my attempts.

Try using "wget", it's faster if you don't need to set up subversion for anything.

And no, avatars are not working.

I'm thinking of just writing a Python script to take care of the bits and bobs that the merge isn't quite working with yet, especially since I have a lot of things I need to move over that won't (indeed, shouldn't!) be handled by the merge system (e.g. custom gallery stuff).

Avatars are not supposed to work. The svn version isn't 100% working for SMF yet. I'm still making changes to correct it. Please be patient.

Also, CaptOblivious, can you please update the Bug #1156 With feedback regarding the changes you suggest to it. I'll look into that when I sit down in my next development session. Please make sure you reference you post (use the link you can get from right clicking your post number) here in this thread in your feedback.

Thanks.

Your changes to the mime_content_type() calls are incorrect, vastly incorrect. You only need to supply a filename, not the full path to the file.

<?php
echo mime_content_type('php.gif') . "\n";
echo mime_content_type('test.php');
?>

Though, this is going to be changed soon anyways since this function has been deprecated.

Wow, I had been under the impressions that mime_content_type actually looked at the file's contents, and so needed a full path. My mistake!

I see you already updated the bug report, so I'll leave it at that.


And my apologies if I appear impatient. It is a little hard to tell what has been omitted by mistake (i.e., mixed entries in the attachments table), and what has been omitted by choice (i.e., avatars haven't been completed yet) in the merge system, so I naturally assume the least possible charitable interpretation…I'll not do that going forward