TweetFollow Us on Twitter

Data Conversion
Volume Number:4
Issue Number:5
Column Tag:Basic School

Quick & Dirty Data Conversion

By Dave Kelly, MacTutor Editorial Board

FILE IMPORT/EXPORT with BASIC

Over the years, one of the most beneficial uses for Basic is the ability to quickly write programs to import/export data to and from data bases. This also provides a “quick and dirty” way to convert data or suppress unwanted characters. I’d like to explain/complain about the process and some of the problems that I have encountered.

My first major data conversion came a couple of years ago when I converted 30,000 names in a mail list from my Apple //e to the Macintosh. Yes, this was quite a feat, mostly because of the bottle neck which the dinky 143K floppy drives allowed on the Apple //. And you thought the Mac 400K drives were small! Anyway, on the Apple //e I was using DB MASTER, a now orphaned product, with some small support still available from a consulting firm in Northern California. There may be some of you that are familiar with DB MASTER (sometimes it was referred to as DB MONSTER). DB MASTER was great for mail lists and still would be today except for it’s slow speed (that’s partly the Apple //s fault).

The process went something like this: First the data needed to be converted from DB MASTER file format to ASCII so that I could direct connect the Mac and the //e together. DB MASTER utilities includes an export utility which would convert files to either DIF or a “One Field per Line” format. The utilities converted my mail list from my Sider hard disk to about 50 143K floppy disks (UGH!). Next the ASCII needed to be converted from Apple // ASCII to ”the rest of the world” ASCII format. ASCII EXPRESS telecommunications software provides a conversion utility for doing this (If they didn’t I’d have to use my own Basic program to convert it myself. Next each disk was converted via direct null modem connection to the Mac. Whew!

To make a long story (and many hours) short, the mail list was now on the Mac, but somehow during the process some control characters were intermixed in the files. Omnis 3 would not read the files with these control characters so it was BASIC to the rescue. There are 128 standard ASCII characters of which the first 31 characters are control characters, traditionally used for teletype terminals. The problem was easily solved by writing a Basic routine to read in the file, filter out the bad characters and rewrite the file. Compile the program to speed it up!

Control Character Filter
‘ ©MacTutor, 1988
‘ By Dave Kelly
‘ MSBASIC

x$=FILES$(1,”TEXT”)
IF x$=”” THEN END
OPEN x$ FOR INPUT AS #1
OPEN “Filter Output” FOR OUTPUT AS #2
WHILE NOT EOF(1)
    Char$=INPUT$(1,1)
    IF ASC(Char$)>=32 THEN PRINT #2,Char$;
    IF Char$=CHR$(13) THEN PRINT #2,Char$;
WEND
CLOSE #1
CLOSE #2
END

That’s ok for simple filtering. Things may need to be more complex if the original file contains unexpected characters in it. Recently, I came across the need to convert data stored in a FileMaker Plus file in order to do some simple, but unusual calculations. As drafting is done for test equipment the progress is logged into a FileMaker Plus file. Each drawing number is entered with its own charge number and the date is entered indicating the start or completion of stages of the drawing process. Occasionally the boss (who is always right because he is the boss) wants a drawing status report according to charge number (there are multiple drawings per charge number). FileMaker Plus will give the total number of drawings but only after searching for each of the charge numbers and dates individually. BASIC to the rescue.

The Drawing Count program was written to count progress of each of the charge numbers. A file is opened for input and another file for output. Each field was previously exported to a BASIC formatted file and is read by the Basic program in the same order which it was written. The charge number is saved, but each of the date fields is replaced by a flag (0 or 1) which indicated if there was anything in the date field. These 0s and 1s are added together to get the totals after the data is sorted. The sort routine could have been beefed up but, for a “quick and dirty” program, it does the job. The program was compiled with the MS Basic compiler which provided adequate speed results.

If you are wondering why ZBasic wasn’t used here, so am I. Actually, I ran into a small problem which appears to be a bug with ZBasic 4.01 while reading the FileMaker Plus exported text files. Although ZBasic’s INPUT# statement is supposed to read data from the file until a carriage return, <COMMA>, End-Of-File, or 255 characters are encountered, when ZBasic saw a comma followed by a carriage return, the null between the comma and carriage return was skipped and the next field (from the next record) was read in. The data looks like this:

“ITEM1”,”ITEM2",”ITEM3",
“ITEM4”,”ITEM5",”ITEM6",”ITEM7"

The null field after item3 was skipped and ITEM4 is read in. This is how data is formatted when dumped in comma delimited format as with FileMaker Plus. Sorry, Zedcor.

COORDINATE WINDOW
DEF OPEN “TEXTDAVE”
BREAK ON
filename$=FILES$(1,”TEXT”,,vol%)
IF filename$=”” THEN END
OPEN “I”,#1,filename$,,vol%
INPUT #1,A$,B$,C$,D$
PRINT @(1,10)A$,B$,C$,D$
CLOSE 1

Another file conversion problem to deal with involves the conversion of text which has been dumped from text fields. FileMaker Plus, for example, inserts an end of line mark at the end of each line when it exports data to a file. When the data is read into another program which doesn’t use the end of line character to mark the end of lines (because the text is word wrapped and reformattable). If you read the FileMaker text field into Double Helix II for example, the end of line character shows up in the text field as an undefined character (a box). The filter method explained above can find these characters if you know what character it is.

Fortunately life is much easier for converting data these days than it was even five years ago. Now I use file formats like DIF and SYLK whenever I can when converting between programs that support both formats. The transfer goes a lot less painfully when the format is well defined as the DIF format is. For those not familiar with DIF, it is called the “Data Interchange Format” and was originally designed to by Software Arts (are they still around?) to transfer data from Visicalc (the father of all spreadsheet programs). The file is completely ASCII but has header information added to help maintain data integrity. (This saved me a few times). This means that the file will undoubtedly be larger than without DIF formatting so be sure that you only use it for file transport rather than data storage especially if your disk free space is critical. SYLK is Microsoft’s format. Most Microsoft products and a few data base programs such as Omnis 3 and FileMaker Plus support SYLK format. The best way to figure out how these work is to analyze some sample data from a program that exports in one of these formats.

‘ Drawing Count Program
‘ Converts Drawing Log Data created by FileMaker Plus
‘ and computes total number of drawings under each date
‘ By Dave Kelly
‘ ©1988
‘MS BASIC

DIM Charge$(1000),Date1(1000),Date2(1000),Date3(1000), Date4(1000)
DIM tCharge$(20),tDate1(20),tDate2(20),tDate3(20),tDate4(20)
filename$=FILES$(1,”TEXT”)
IF filename$=”” THEN END
OPEN “I”,1,filename$
OPEN “O”,2,”Drawing Count.DATA”
PRINT
count=0
CALL MOVETO(10,30)
PRINT “Now Reading in Records...”
WHILE NOT EOF(1)
    count=count+1
    CALL MOVETO (10,50)
    PRINT count
    Charge$(count)=””
    i$=””
    A$=””
    C$=””
    S$=””
    INPUT#1,Charge$(count),i$,A$,C$,S$
    IF LEN(Charge$(count))>0 THEN
        IF RIGHT$(Charge$(count),1)=”)” THEN
            Charge$(count)=MID$(Charge$(count),LEN(Charge$(count))-4,4)
        ELSE
            Charge$(count)=RIGHT$(Charge$(count),4)
        END IF
    END IF
    IF LEN(i$)>0 THEN Date1(count)=1 ELSE Date1(count)=0
    IF LEN(A$)>0 THEN Date2(count)=1 ELSE Date2(count)=0
    IF LEN(C$)>0 THEN Date3(count)=1 ELSE Date3(count)=0
    IF LEN(S$)>0 THEN Date4(count)=1 ELSE Date4(count)=0
WEND
CALL MOVETO(10,70)
PRINT “Now Sorting Data...”
CALL Sort(count,Charge$(),Date1(),Date2(),Date3(),Date4())
k=0
FOR j=1 TO count
IF tCharge$(k)<>Charge$(j) THEN k=k+1
tCharge$(k)=Charge$(j)
tDate1(k)=tDate1(k)+Date1(j)
tDate2(k)=tDate2(k)+Date2(j)
tDate3(k)=tDate3(k)+Date3(j)
tDate4(k)=tDate4(k)+Date4(j)
NEXT j
CALL MOVETO(10,90)
PRINT “Now Writing to File...”
PRINT#2,”Charge#    Date1    Date2    Date3    Date4"
FOR i=1 TO k
tDate1(i)=tDate1(i)-tDate2(i)
tDate2(i)=tDate2(i)-tDate3(i)
tDate3(i)=tDate3(i)-tDate4(i)
PRINT #2,tCharge$(i),tDate1(i),tDate2(i),tDate3(i),tDate4(i)
PRINT tCharge$(i),tDate1(i),tDate2(i),tDate3(i),tDate4(i)
NEXT i
CLOSE #1
CLOSE #2
PRINT”Click to continue...”
WHILE MOUSE(0)<>1:WEND
END

SUB Sort(count,Charge$(),Date1(),Date2(),Date3(),Date4()) STATIC
    flips=1
    WHILE flips
        flips=0
        FOR i=2 TO count
            IF Charge$(i-1)> Charge$(i) THEN
                flips=1
                SWAP Charge$(i-1),Charge$(i)
                SWAP Date1(i-1),Date1(i)
                SWAP Date2(i-1),Date2(i)
                SWAP Date3(i-1),Date3(i)
                SWAP Date4(i-1),Date4(i)
            END IF
        NEXT
    WEND
END SUB

HYPER-CORNER

The new update disk for HyperCard version 1.1 contains a read me stack with three new buttons pertaining to importing and exporting data. If you don’t have the new update you can get it from your Apple dealer. Version 1.1 is supposed to fix some bugs that “bugged” some of us.

The read me stack gives directions for using the new buttons, but I’ll try to explain them a little bit here. To try them out, create a new stack with a few fields in it such as the one I created; see fig. 1.

Fig 1. Hyper Dialog

HyperCard Import/Export

The arrow keys were added from the Button Ideas stack to make it easy to check out the data, but you will need to remember the command 1,2,3,4 keys that move you from card to card or you may use the arrow (or option-arrow) keys to move from field to field.

Fig. 2 Export Dialog

The most useful of the buttons is the Export Data button. It allows you to select the background fields you wish to export and the delimiters separating the fields and records. Most database programs will read the tab delimited records (with carriage return between each record). NO, the export button will not export card fields, only background fields. That is understandable since the output needs to be consistent.

I found that the best way to see how the buttons work is to try them out. Clicking on the Export Data (or Export Text) button will produce this dialog which will let you select the background field you desire to output. (See figure 2)

Selecting the Fields for Export

After selecting the All button or after selecting which of the background fields you want to export, you specify the file name to export to. A dialog comes up to ask you for the name of the text file to export to:

Fig. 3 Editable Dialog

Selecting a Filename for Export

There is an XCMD around someplace for calling the Standard File Dialog which would be better to substitute for this dialog. The problem with XCMD routines for general use is that not everyone has them. It would be best to set up your own custom library of XCMDs to use in your own stacks.

I ran into no difficulty getting my data to export to a file. You can examine your data with a word processor to see how the text file looks when the data has been exported. I like using MS Word for examining this kind of file because you can view all the tabs and carriage returns using the Show ¶ menu item.

The Import Button works well too, except that you can’t import data into existing background fields. It creates the background fields on the fly. Perhaps someone out there could create a button script that will allow you to specify which field you would like to be the first, second and so on field to be read in. AT LEAST, now these buttons are included with HyperCard. At first, it may appear that the import didn’t work, but if you read the instructions in the read me stack, it explains that the new background fields are created one on top of another. To see the fields you will have to peel them apart. (see figure below)

Imported Fields

Ideally, the importing will be done first thing before the stack layout is designed if possible. Importing/Exporting of text with HyperCard has been one of the weak links. With all the interest in HyperCard these days, there should be more XCMD routines and tools we can use to improve the usefulness of HyperCard.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Dragon Dictate 6.0 - Premium voice-recog...
With Dragon Dictate speech recognition software, you can use your voice to create and edit text or interact with your favorite Mac applications. Far more than just speech-to-text, Dragon Dictate lets... Read more
OmniFocus 3.11.7 - GTD task manager with...
OmniFocus is an organizer app. It uses projects to organize tasks naturally, and then add tags to organize across projects. Easily enter tasks when you’re on the go, and process them when you have... Read more
rekordbox 6.5.1.0009 - Professional DJ m...
rekordbox is the best way of preparing and managing your tracks, be it at home, in the studio, or even on the plane! It allows you to import music from other music-management software using the... Read more
1Password 7.8.1 - Powerful password mana...
1Password is a password manager that uniquely brings you both security and convenience. It is the only program that provides anti-phishing protection and goes beyond password management by adding Web... Read more
Ableton Live 10.1.35 - Record music usin...
Ableton Live lets you create and record music on your Mac. Use digital instruments, pre-recorded sounds, and sampled loops to arrange, produce, and perform your music like never before. Ableton Live... Read more
Microsoft Office 365, 2019 16.48 - Popul...
Microsoft Office 365. The essentials to get it all done. Unmistakably Office, designed for Mac Get started quickly with new, modern versions of Word, Excel, PowerPoint, Outlook and OneNote-... Read more
Adobe After Effects 18.1 - Create profes...
After Effects is available as part of Adobe Creative Cloud for $52.99/month (or $20.99/month for a single app license). The new, more connected After Effects can make the impossible possible. Get... Read more
Adobe Audition 14.1 - Professional post-...
Audition is available as part of Adobe Creative Cloud for as little as $20.99/month (or $9.99/month if you're a previous Audition customer). Adobe Audition empowers you to create and deliver... Read more
Adobe Animate 21.0.5 - Animation authori...
Animate is available as part of Adobe Creative Cloud for as little as $20.99/month (or $9.99/month if you're a previous Flash Professional customer). Animate (was Flash CC) lets you share work... Read more
Adobe Photoshop 22.3.1 - Professional im...
You can download Photoshop for Mac as a part of Creative Cloud for only $20.99/month (or $9.99/month if you have purchased an earlier software version). Adobe Photoshop is a recognized classic of... Read more

Latest Forum Discussions

See All

Moonlight Sculptor is an upcoming MMORPG...
Kakao Games and XL Games – who you might be familiar with from their previous game ArcheAge – have announced that their MMORPG Moonlight Sculptor is now available to pre-order for iOS and Android devices. Moonlight Sculptor has previously launched... | Read more »
MU Archangel is now open for pre-registr...
MU Archangel is now open for pre-registration in Southeast Asia following its massive success in other territories. Players from Singapore, Thailand, Malaysia, Indonesia, and the Philippines (except Vietnam) can now join in on the fun by applying... | Read more »
Compete, a new social media app you can...
Whoever told you you can’t get rich making videos has obviously never heard of Compete, Competitive Media Technologies Limited’s hot new social media app where you can rake in all the dough just by doing what you love. Video monetization that... | Read more »
Bethesda has released a new DOOM mobile...
Bethesda Softworks has released a new DOOM game out of the blue exclusively for mobile devices. It’s called Mighty DOOM and is currently only available as an early access title on Android but will be expanding to more users in the future. [Read... | Read more »
Anagraphs is a word puzzle game with a t...
Cinq-Mars Media has released its word puzzle game Anagraphs for iOS and Android devices. The game released last week after a short delay in getting it onto the appropriate platforms. [Read more] | Read more »
These are the top 5 best iPhone games li...
Fortnite has been the big hitter in mobile gaming this year, and it's not hard to see why. Thanks to some excellent marketing, and a polished experience that almost anyone can enjoy, it's really taken the App Store by storm. But there are other... | Read more »
The top 5 best iPhone games like Pokemon...
Pokemon GO is still the, if you'll excuse the pun, go-to game if you want some AR action on your phone. But it's not the only choice out there, and if you've got a hankering for something a bit different, then your eyes might already have started... | Read more »
The top 5 best iPhone games like Starcra...
Starcraft sits at the top of the RTS tree for a number of very good reasons. It also isn't on mobile, again, for a number of very good reasons. But that doesn't mean you can't find a way to indulge your sci-fi, competitive, massive, or engaging RTS... | Read more »
Apple Arcade: Ranked - Top 25 [Updated 4...
In case you missed it, I am on a quest to rank every Apple Arcade game there is. [Read more] | Read more »
The top 5 best iPhone games like The Roo...
The Room has had a massive impact on the world of mobile gaming. Not only is it a brilliant adventure, it also shows how the touchscreen controls on your iPhone can be turned into something far more elegant and tactile than just a bunch of buttons... | Read more »

Price Scanner via MacPrices.net

B&H is offering clearance prices on lefto...
Apple reseller B&H Photo has clearance 2020 13″ 1.4GHz Intel-based MacBook Pros on sale today for $200-$300 off Apple’s original MSRP with prices starting at only $1099. Expedited shipping is... Read more
Roundup of Today’s Best MacBook Deals: M1 Mac...
Apple resellers are offering sale prices on Apple’s M1-powered 13″ MacBook Airs ranging up to $190 off MSRP. Here’s where to pick one up today, and as always, keep an eye on our 13″ MacBook Air Price... Read more
Apple AirPods Pro drop to new low price of on...
Amazon has Apple’s AirPods Pro on sale today for a new low price of only $197 shipped. That’s $52 off MSRP and the lowest price currently available for a set of AirPods Pro from any Apple reseller.... Read more
Apple restocks clearance 13″ Intel-based MacB...
Apple has clearance, Certified Refurbished, 2020 13″ Intel-based MacBook Airs available starting at only $809 and up to $280 off original MSRP. Each MacBook features a new outer case, comes with a... Read more
OWC drops prices on 2020 Intel multi-core Mac...
Other World Computing has clearance 2020 Intel-based Mac minis on sale starting at only $499. Both 4-core and 6-core models are in stock today. These are new, unopened, factory-sealed minis: – 3.6GHz... Read more
Save $50 off Apple’s 10.9″ iPad Air today at...
B&H Photo has new 10.9″ Apple iPad Airs in stock and on sale today for up to $50 off MSRP. Expedited shipping is free to most addresses in the US. Note that some sale prices may be restricted to... Read more
Rare Apple sale: Get a HomePod mini for $10 o...
Apple reseller Expercom has the Space Gray HomePod mini on sale today for $89 shipped. Their price is $10 off Apple’s MSRP, and it’s currently the only sale price available for a HomePod mini among... Read more
Apple has M1 Mac minis available starting at...
Apple has a full line of standard configuration M1 Mac minis available in their Certified Refurbished section starting at only $589 and up to $140 off MSRP. Each mini comes with Apple’s one-year... Read more
New sale at Amazon: $55-$70 discounts on Appl...
Amazon has 7.9″ iPad minis on sale today for up to $70 off Apple’s MSRP, each including free shipping. Prices start at $344. These are the same iPad minis sold by Apple in their retail and online... Read more
Apple offering 13″ M1 MacBook Airs for as lit...
Apple has a full line of 2020 13″ M1 MacBook Airs available, Certified Refurbished, starting at only $849 and up to $190 off original MSRP. These are the cheapest M1 MacBook Airs for sale today at... Read more

Jobs Board

*Apple* Valley 20hr Teller - Wells Fargo (Un...
…or scheduled + Ability to stand for extended periods of time **Street Address** **MN- Apple Valley:** 14325 Cedar Ave - Apple Valley, MN **Disclaimer** All offers Read more
*Apple* Valley 20hr Teller - Wells Fargo (Un...
…or scheduled + Ability to stand for extended periods of time **Street Address** **MN- Apple Valley:** 14325 Cedar Ave - Apple Valley, MN **Disclaimer** All offers Read more
Desktop Support Technician - *Apple* / Mac...
…infrastructure and internal desktop systems. Must have an IT background that includes Apple / Mac support. **Overview:** + Responds to routine technical questions or Read more
Geek Squad *Apple* Consultation Professiona...
**801042BR** **Job Title:** Geek Squad Apple Consultation Professional **Job Category:** Store Associates **Store Number or Department:** 000214-Willowbrook-Store Read more
*Apple* Mobility Specialist - Best Buy (Unit...
**800879BR** **Job Title:** Apple Mobility Specialist **Job Category:** Store Associates **Store Number or Department:** 000803-Lansing-Store **Job Description:** Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.